Uncertainty and Sensitivity Analysis Methods for Improving Design Robustness and Reliability Qinxian He

Home , Sensitivity analysis

Uncertainty and Sensitivity Analysis Methods for Improving Design Robustness and Reliability by Qinxian He Bachelor of Science in Engineering, Duke University, 2008 Master of Science, Massachusetts Institute of Technology, 2010 Submitted to the Department of Aeronautics and Astronautics in partial fulﬁllment of the requirements for the degree of Doctor of Philosophy at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY June 2014 c Massachusetts Institute of Technology 2014. All rights reserved.

Author...... Department of Aeronautics and Astronautics May 7, 2014 Certified by...... Karen E. Willcox Professor of Aeronautics and Astronautics Thesis Supervisor Certified by...... Ian A. Waitz Jerome C. Hunsaker Professor of Aeronautics and Astronautics Committee Member Certified by...... Douglas L. Allaire Assistant Professor of Mechanical Engineering, Texas A&M University Committee Member Accepted by ...... Paulo C. Lozano Associate Professor of Aeronautics and Astronautics Chair, Graduate Program Committee 2 Uncertainty and Sensitivity Analysis Methods for Improving Design Robustness and Reliability by Qinxian He

Submitted to the Department of Aeronautics and Astronautics on May 7, 2014, in partial fulﬁllment of the requirements for the degree of Doctor of Philosophy

Abstract

Engineering systems of the modern day are increasingly complex, often involving numerous components, countless mathematical models, and large, globally-distributed design teams. These features all contribute uncertainty to the system design process that, if not properly managed, can escalate into risks that seriously jeopardize the design program. In fact, recent history is replete with examples of major design setbacks due to failure to recognize and reduce risks associated with performance, cost, and schedule as they emerge during the design process. The objective of this thesis is to develop methods that help quantify, understand, and mitigate the effects of uncertainty in the design of engineering systems. The design process is viewed as a stochastic estimation problem in which the level of uncertainty in the design parameters and quantities of interest is characterized probabilistically, and updated through successive iterations as new information becomes available. Proposed quantitative measures of complexity and risk can be used in the design context to rigorously estimate uncertainty, and have direct implications for system robustness and reliability. New local sensitivity analysis techniques facilitate the approximation of complexity and risk in the quantities of interest resulting from modifications in the mean or variance of the design parameters. A novel complexity- based sensitivity analysis method enables the apportionment of output uncertainty into contributions not only due to the variance of input factors and their interactions, but also due to properties of the underlying probability distributions such as intrinsic extent and non-Gaussianity. Furthermore, uncertainty and sensitivity information are combined to identify specific strategies for uncertainty mitigation and visualize tradeoffs between available options. These approaches are integrated with design budgets to guide decisions regarding the allocation of resources toward improving system robustness and reliability.

3 The methods developed in this work are applicable to a wide variety of engineering systems. In this thesis, they are demonstrated on a real-world aviation case study to assess the net cost-benefit of a set of aircraft noise stringency options. This study reveals that uncertainties in the scientific inputs of the noise monetization model are overshadowed by those in the scenario inputs, and identifies policy implementation cost as the largest driver of uncertainty in the system.

Thesis Supervisor: Karen E. Willcox Title: Professor of Aeronautics and Astronautics

Committee Member: Ian A. Waitz Title: Jerome C. Hunsaker Professor of Aeronautics and Astronautics

Committee Member: Douglas L. Allaire Title: Assistant Professor of Mechanical Engineering, Texas A&M University

4 Acknowledgments

The journey toward a PhD is one that cannot be undertaken alone. It is only with help from countless people along the way that I have made it thus far, and I am eternally grateful for their support. First and foremost, I would like to thank my advisor, Professor Karen Willcox, for her guidance, mentorship, and encouragement over the past four years. It has truly been a privilege to work with Karen, and I am greatly indebted to her for everything she has taught me about research, life, and making it all balance together. I also extend thanks to my committee members, Dean Ian Waitz and Professor Doug Allaire, for lending their knowledge and support to every aspect of this research, and for providing invaluable feedback and guidance along the way. I would also like to acknowledge my thesis readers, Professor John Deyst and Professor Olivier de Weck for contributing insightful comments and suggestions that have helped to improve this thesis.

To my labmates in the ACDL, past and present, thank you so much for sharing this journey with me. It has been an honor to work with and learn from all of you. I would like to especially recognize Leo Ng, Andrew March, Sergio Amaral, Chad Lieberman, Giulia Pantalone, Harriet Li, Laura Mainini, R´emi Lam, Marc Lecerf, Tiangang Cui, Patrick Blonigan, Alessio Spantini, Eric Dow, Hemant Chaurasia, and Xun Huan for making my stay in Building 37 such a memorable experience. I am also grateful to the larger AeroAstro community for providing a welcoming and dynamic environment in which to learn and grow. I have made so many friends in AeroAstro and will always fondly remember the good times we’ve shared. A special thank you to the ladies of the Women’s Graduate Association of Aeronautics and Astronautics (WGA3) — Sunny Wicks, Sameera Ponda, Farah Alibay, Whitney Lohmeyer, Abhi Butchibabu, Sathya Silva, and Carla Perez Martinez — it has been such a joy getting to know you and I can’t wait to see the amazing things you will continue to accomplish in the future. I also owe a great deal of thanks to Philip Wolfe. From a technical standpoint, thank you for helping me set up and run the CAEP/9 noise stringency case study on such short notice; from a personal perspective, thank you for your

5 friendship over the past 10 years — I have really appreciated our project meetings, commiseration lunches, and random adventures both at Duke and at MIT. Finally, I would like to acknowledge the AeroAstro staﬀ members who work tirelessly to make life easier and more enjoyable for graduate students. Jean Sofronas, Joyce Light, Meghan Pepin, Marie Stuppard, Beth Marois, Bill Litant, Robin Palazzolo, and Sue Whitehead — thank you for being so generous with your time, help, smiles, and words of encouragement.

Outside of research, I have been extremely fortunate to have the support of a diverse group of friends who help me keep things in perspective. Matt and Amanda Edwards, Jim Abshire, David Rosen, Emily Kim, Audrey Fan, Lori Ling, Yvonne Yamanaka, Tiffany Chen, Tatyana Shatova, and Robin Chisnell are fellow graduate students in the Boston area, and dear friends who have accompanied me on this journey. Thank you for the potluck dinners, ski trips, hiking outings, gym workouts, and runs along the Charles River — you have done so much to lift my spirits and bring joy to my life. Another huge source of strength comes from the Sidney-Pacific Graduate Community, without a doubt the best graduate dormitory in the world. Living at SP was one of the highlights of my time at MIT, and has taught me so much about friendship, leadership, and the spirit of community. In particular, I wish to thank the former Housemaster team — Professor Roger Mark and Mrs. Dottie Mark, and Professor Annette Kim and Dr. Roland Tang — thank you for welcoming me into your home and making sure that I am always well-fed. To my friends from SP — Amy Bilton, Mirna Slim, Ahmed Helal, Jit Hin Tan, Brian Spatocco, George Lan, Po-Ru Loh, Kendall Nowocin, George Chen, Boris Braverman, FabiánKozynski, and many, many others — thank you for sharing your kindness and talents, for inspiring me to set ambitious goals, and for showing me that there are few problems in life that can’t be solved through teamwork and more bacon.

Finally, I would like to thank my family for their steadfast love and support. My parents, Jie Chen and Helin He, left behind friends, family, and successful careers in China to immigrate to the United States in pursuit of better opportunities. Without their sacriﬁce, I would not be where I am today. I credit them for instilling in me the

6 importance of a good education, the determination to overcome whatever challenges may arise, and the conﬁdence to forge my own path. Last but not least, I want to thank Tim Curran for being my co-pilot on this incredible journey. Thank you for always believing in me, for teaching me to be patient with myself and others, and for giving me the strength to push through the tough times. I look forward to starting the next chapter of our lives together. The work presented in this thesis was funded in part by the International De- sign Center at the Singapore University of Technology and Design, by the DARPA META program through AFRL Contract Number FA8650-10-C-7083 and Vanderbilt University Contract Number VU-DSR #21807-S7, by the National Science Founda- tion Graduate Research Fellowship Program, and by the Zonta International Amelia Earhart Fellowship Program.

Contents

1 Introduction 19 1.1 Motivation for Uncertainty Quantiﬁcation ...... 20 1.2 Terminology and Scope ...... 21 1.2.1 System Design Process ...... 21 1.2.2 Characterizing Uncertainty ...... 24 1.2.3 Some Common Probability Distributions ...... 26 1.3 Designing for Robustness and Reliability ...... 28 1.3.1 Deﬁning Robustness and Reliability ...... 29 1.3.2 Background and Current Practices ...... 30 1.3.3 Opportunities for Intellectual Contribution ...... 33 1.4 Thesis Objectives ...... 33 1.5 Thesis Outline ...... 34

2 Quantifying Complexity and Risk in System Design 37 2.1 Background and Literature Review ...... 37 2.1.1 Quantitative Complexity Metrics ...... 38 2.1.2 Diﬀerential Entropy ...... 40 2.2 A Proposed Deﬁnition of Complexity ...... 41 2.2.1 Exponential Entropy as a Complexity Metric ...... 42 2.2.2 Interpreting Complexity ...... 44 2.2.3 Numerical Complexity Estimation ...... 46 2.3 Related Quantities ...... 47

9 2.3.1 Variance ...... 49 2.3.2 Kullback-Leibler Divergence and Cross Entropy ...... 51 2.3.3 Entropy Power ...... 52 2.4 Estimating Risk ...... 53 2.5 Chapter Summary ...... 54

3 Sensitivity Analysis 55 3.1 Background and Literature Review ...... 56 3.1.1 Variance-Based Global Sensitivity Analysis ...... 56 3.1.2 Vary-All-But-One Analysis ...... 59 3.1.3 Distributional Sensitivity Analysis ...... 60 3.1.4 Relative Entropy-Based Sensitivity Analysis ...... 61 3.2 Extending Distributional Sensitivity Analysis to Incorporate Changes in Distribution Family ...... 62 3.2.1 Cases with No Change in Distribution Family ...... 65 3.2.2 Cases Involving Truncated Extent for Xo ...... 67 3.2.3 Cases Involving Resampling ...... 71 3.3 Local Sensitivity Analysis ...... 75 3.3.1 Local Sensitivity to QOI Mean and Standard Deviation . . . . 76 3.3.2 Relationship to Variance-Based Sensitivity Indices ...... 77 3.3.3 Local Sensitivity to Design Parameter Mean and Standard De- viation ...... 79 3.4 Entropy Power-Based Sensitivity Analysis ...... 81 3.4.1 Entropy Power Decomposition ...... 81 3.4.2 The Eﬀect of Distribution Shape ...... 85 3.5 Chapter Summary ...... 88

4 Application to Engineering System Design 91 4.1 R-C Circuit Example ...... 91 4.2 Sensitivity Analysis of the R-C Circuit ...... 94 4.2.1 Identifying the Drivers of Uncertainty ...... 94

10 4.2.2 Understanding the Impacts of Modeling Assumptions . . . . . 94 4.3 Design Budgets and Resource Allocation ...... 101 4.3.1 Visualizing Tradeoﬀs ...... 102 4.3.2 Cost and Uncertainty Budgets ...... 104 4.3.3 Identifying and Evaluating Design Alternatives ...... 107 4.4 Chapter Summary ...... 114

5 Application to a Real-World Aviation Policy Analysis 115 5.1 Background and Problem Overview ...... 115 5.1.1 Aviation Noise Impacts ...... 115 5.1.2 CAEP/9 Noise Stringency Options ...... 117 5.2 APMT-Impacts Noise Module ...... 119 5.2.1 Model Inputs ...... 120 5.2.2 Estimating Monetary Impacts ...... 125 5.3 Results and Discussion ...... 127 5.3.1 Comparison of Stringency Options and Discount Rates . . . . 127 5.3.2 Global and Distributional Sensitivity Analysis ...... 128 5.3.3 Visualizing Tradeoﬀs ...... 132 5.4 Chapter Summary ...... 135

6 Conclusions and Future Work 137 6.1 Thesis Summary ...... 137 6.2 Thesis Contributions ...... 138 6.3 Future Work ...... 139

A Bimodal Distribution with Uniform Peaks 141 A.1 Specifying Distribution Parameters ...... 142 A.2 Options for Variance Reduction ...... 143 A.3 Switching to a Bimodal Uniform Distribution ...... 145

B Derivation of Local Sensitivity Analysis Results 147 B.1 Perturbations in Mean ...... 147

11 B.1.1 Effect on Complexity ...... 148 B.1.2 Effect on Risk ...... 148 B.2 Perturbations in Standard Deviation ...... 149 B.2.1 Effect on Complexity ...... 150 B.2.2 Effect on Risk ...... 150

C Derivation of the Entropy Power Decomposition 153

D CAEP/9 Analysis Airports 157

Bibliography 161

12 List of Figures

1-1 The eﬀect of complexity on system development time in the aerospace, automobile, and integrated circuit industries [1, Figure 1] ...... 20

1-2 Schematic of the proposed system design process ...... 22

1-3 Examples of uniform, triangular, Gaussian, and bimodal uniform probability densities ...... 26

1-4 Schematics of design for robustness and reliability. Red and blue prob-

ability densities represent initial and updated estimates of fY (y), respectively. Dashed line indicates the location of the requirement r. Shaded portions denote regions of failure in which the requirement is not satisﬁed...... 29

2-1 Probability density of a standard Gaussian random variable (left) and a uniform random variable of equal complexity (right) ...... 45

2-2 Probability density of a multivariate Gaussian random variable with

µ = (0, 0) and Σ = I2 (top left), and uniform random variables of equal complexity in R2 and R (top right and bottom, respectively) . . . . . 45 2-3 Numerical approximation of h(Y ) using k bins of equal size ...... 47

2-4 Comparison of variance, diﬀerential entropy, and complexity between a bimodal uniform distribution and its equivalent Gaussian distribution 50

2-5 Probability of failure associated with the requirement r ...... 53

3-1 Apportionment of output variance in GSA [8, Figure 3-1] ...... 56

13 3-2 Examples of reasonable uniform, triangular, and Gaussian distribu-

tions. In each ﬁgure, the solid red line denotes fXo (x), the dashed blue

lines represent various possibilities for fX0 (x), and the dotted black

lines show κfXo (x), which is greater than or equal to the correspond-

ing density fX0 (x) for all x...... 65 3-3 Examples of reasonable uniform, triangular, and Gaussian distributions for cases involving a truncated extent for Xo ...... 69 3-4 Examples of reasonable uniform, triangular, and Gaussian distributions for cases involving resampling. The left column shows instances where δ is too large, and resampling is required. The right column shows instances where δ is suﬃciently small, and AR sampling can be used to generate samples of X0 from samples of Xo...... 73

3-5 The relative locations of r and µY greatly impact the change in risk

associated with a decrease in σY . Moving from the red probability

density to the blue, P (Y < r) decreases if µY − r > 0, and increases if

µY − r < 0...... 77

3-6 Examples of Y = X1 + X2 with increase, decrease, and no change in Gaussianity between the design parameters and QOI ...... 87

3-7 Sensitivity indices Si, ηi, and ζi for three examples of Y = X1 + X2.

For each factor, Si equals the product of ηi and ζi...... 88

4-1 R-C high-pass filter circuit ...... 92 4-2 Bode diagrams for magnitude (top) and phase (bottom). The dashed black line indicates the required cutoff frequency of 300 Hz...... 92 4-3 Histogram of cutoff frequency for Case 0 generated using 10,000 MC samples. The dashed black line indicates the required cutoff frequency of 300 Hz...... 93 4-4 Global and distributional sensitivity analysis results for Case 0 . . . . 94 4-5 Distributional sensitivity analysis results with changes to distribution family ...... 95

14 4-6 Examples of uniform (U), triangular (T), Gaussian (G), and bimodal 2 uniform (BU) distributions for X2, each with a variance of 0.30 µF . 96 4-7 Examples of uniform (U), triangular (T), Gaussian (G), and bimodal

uniform (BU) distributions for X2 supported on the interval [3.76, 5.64] µF 98

4-8 Histogram of Y resulting from various distributions for X2 considered in Case 2 ...... 99 4-9 Main eﬀect and entropy power-based sensitivity indices for Case 2 . . 100 4-10 Contours for variance, complexity, and risk corresponding to reductions in factor variance, generated from 10,000 MC simulations (solid colored lines) or approximated using distributional or local sensitivity analysis results (dashed black lines) ...... 103 4-11 Notional cost associated with a 100(1 − δ)% reduction in the variance

of X1 or X2 ...... 105 4-12 Contours for variance, complexity, and risk corresponding to reductions in factor variance (solid colored lines) overlaid with contours for cost of implementation (dashed green lines) ...... 106 4-13 Contours for risk (solid blue lines) overlaid with contours for cost (dashed green lines). Shaded region denotes area in which all budget constraints are satisfied...... 107 4-14 Uncertainty contours for variations in resistance and capacitance, generated from 10,000 MC simulations (solid colored lines) or approximated using entropy power-based sensitivity analysis results (dashed black lines) ...... 108 4-15 Uncertainty contours for variations in resistance and resistor tolerance 108 4-16 Uncertainty contours for variations in capacitance and capacitor tolerance109 4-17 Uncertainty contours for variations in resistor and capacitor tolerance 109 4-18 Tradeoffs in complexity and risk for various design options. Dashed green lines bound region where both complexity and risk constraints are satisfied...... 113

15 5-1 Joint and marginal distributions for the regression parameters obtained using 10,000 bootstrapped samples ...... 122 5-2 Schematic of APMT-Impacts Noise Module [142, adapted from Figure 4]126 5-3 Comparison of mean estimates for Y in Stringencies 1–3 and discount rates between 3%–9%. Error bars denote 10th, 90th percentile values. 127 5-4 Main eﬀect sensitivity indices for Stringency 1, DR = 3%, with IC modeled probabilistically in (a) and as a deterministic value in (b) . . 129 5-5 Adjusted main eﬀect sensitivity indices for Stringency 1, DR = 3%, with IC modeled probabilistically in (a) and as a deterministic value in(b)...... 130 5-6 Histogram of Y under various IC assumptions for Stringency 1, DR = 3%...... 131 5-7 Uncertainty contours associated with changes in the mean and variance of BNL and IC, generated from 10,000 MC simulations (solid colored lines) or approximated using sensitivity analysis results (dashed black lines) ...... 132 5-8 Uncertainty contours associated with changes in the mean and variance of CU and IC ...... 133 5-9 Uncertainty contours associated with changes in the mean and variance of BNL and CU ...... 134

A-1 Bimodal uniform distribution with peak bounds aL, bL, aR, and bR,

and peak heights fL and fR ...... 141 A-2 Examples of distributions for X0 obtained using Algorithms 1–3 (solid lines), based on an initial distribution XBU o (dashed green line) and δ = 0.5...... 145 A-3 Examples of switching from a uniform, triangular, or Gaussian distribution to a bimodal uniform distribution. In each ﬁgure, the solid

red line denotes fXo (x), the dashed blue line represents fX0 (x), and

the dotted black line shows κfXo (x), which is greater than or equal to

fX0 (x) for all x...... 146

16 List of Tables

2.1 Range, mean, variance, diﬀerential entropy, exponential entropy, and entropy power for distributions that are uniform, triangular, Gaussian, or bimodal with uniform peaks ...... 48

2.2 The properties of variance, diﬀerential entropy, exponential entropy, and entropy power under constant scaling or shift ...... 49

var (X0) 3.1 Ratio of δ = var (Xo) for changes in factor distribution between the uniform, triangular, and Gaussian families. Rows represent the initial distribution for Xo and columns correspond to the updated distribution for X0...... 64

3.2 Local sensitivity analysis results for complexity and risk ...... 76

4.1 Change in uncertainty estimates for Cases 1-T, 1-G, and 1-BU, as compared to Case 0 ...... 96

4.2 Change in uncertainty estimates for Cases 2-T, 2-G, and 2-BU, as compared to Case 0 ...... 98

4.3 Best achievable uncertainty mitigation results given individual budgets for cost and uncertainty. Entries in red denote budget violations. . . . 107

4.4 Local sensitivity predictions for the change in mean and standard deviation (SD) of each factor required to reduce risk to 10% ...... 112

4.5 Uncertainty and cost estimates associated with various design options. Entries in red denote budget violations...... 113

17 5.1 Inputs to CAEP/9 noise stringency case study. The definitions for BNL, CU, RP, SL, and IGR correspond to the midrange lens of the APMT-Impacts Noise Module, whereas IC and DR are used in the post-processing of impacts estimates to compute net cost-benefit. . . 124 5.2 Comparison of mean and uncertainty estimates of Y for Stringencies 1– 3 and discount rates between 3%–9%. All monetary values (mean, 10th and 90th percentiles, standard deviation, and complexity) are listed in 2006 USD (billions)...... 128 5.3 Main effect sensitivity indices for Stringency 1 at various discount rates 129

D.1 CAEP/9 US airports and income levels ...... 160

18 Chapter 1

Introduction

Over the years, engineering systems have become increasingly complex, with astro- nomical growth in the number of components and their interactions. With this rise in complexity comes a host of new challenges, such as the adequacy of mathematical models to predict system behavior, the expense and time to conduct experimentation and testing, and the management of large, globally-distributed design teams. These issues all contribute uncertainty to the design process, which can have serious implications for system robustness and reliability and potentially disastrous repercussions for program outcome. To address some of these issues, this thesis presents a stochastic process model to describe the design of complex systems, in which uncertainty in various parameters and quantities of interest is characterized probabilistically, and updated through successive design iterations as new estimates become available. Incorporated in the model are methods to quantify system complexity and risk, and reduce them through the allocation of resources to enhance robustness and reliability. The goal of this approach is to enable the rigorous quantification and management of uncertainty, thereby serving to help mitigate technical and programmatic risk. Section 1.1 describes the motivation for uncertainty quantification, followed by Section 1.2, which defines the scope of the research and the terminology used to describe it. Section 1.3 gives an overview of current methodologies used in design for robustness and reliability, and identifies several potential areas for intellectual

19 contribution. Sections 1.4 and 1.5 outline the main research objectives and provide a roadmap for the remainder of the thesis.

1.1 Motivation for Uncertainty Quantiﬁcation

In the aerospace industry, the challenges associated with complexity are particularly daunting. A recent study by the Defense Advanced Research Projects Agency shows that as aerospace systems have become more complex, their associated costs and development times have also reached unsustainable levels (Figure 1-1) [1].1 Until this unmitigated growth can be contained, it poses a looming threat to the continued viability and competitiveness of the industry.

Figure 1-1: The eﬀect of complexity on system development time in the aerospace, automobile, and integrated circuit industries [1, Figure 1]

To understand the potentially devastating eﬀects of complexity in its various manifestations, one needs only to look to recent history, which is replete with examples of major design setbacks due to failure to recognize and reduce performance, cost,

1The complexity metric used in the study is the number of parts plus the source lines of code. For a discussion of various measures of complexity and their applicability, see Section 2.1.

20 and schedule risks as they emerge during the design process. A notable example is the Hubble Space Telescope which, when first launched, failed its optical resolution requirement by an order of magnitude. A Shuttle repair mission, costing billions of additional dollars, was required to remedy the problem [2]. An investigation later uncovered that data revealing optical system errors were available during the fab- rication process, but were not recognized and fully examined; in fact, crucial error indicators were disregarded and key verification steps omitted. The V-22 Osprey tilt- rotor aircraft is another example: over the course of its 25-year development cycle, the program was fraught with safety, reliability, and affordability challenges, resulting in numerous flight test crashes with concomitant loss of crew and passenger lives [3]. More recently, the Boeing 787 Dreamliner transport aircraft program has experi- enced a number of major prototype subsystem test failures, causing budget overruns of billions of dollars and service introduction delays of about three years. Many of the program’s problems were attributed to Boeing’s aggressive strategy to outsource component design and assembly, which created heavy program management burdens and led to unforeseen challenges during vehicle integration [4]. In these cases and numerous others, the design program was unaware of the mount- ing risks in the system, and was surprised by one or more unfortunate outcomes. Al- though these examples are extreme, they are suggestive that improved system design practices are required in order to identify and address performance, cost, and schedule risks as they emerge. Tackling the problem of complexity — and more broadly — of uncertainty in the design of engineering systems is an enormous yet urgent task that requires innovation and collaboration on both sides of industry and academia.

1.2 Terminology and Scope

1.2.1 System Design Process

Figure 1-2 shows a schematic of the system design process proposed in this thesis. It outlines the key steps in the design of engineering systems, by which is meant col-

21 Set Define Generate Evaluate Quantify Perform Targets Parameters Designs Quantities Uncertainty Sensitivity of Interest Analysis 푋1 Models,

Require- Governing 푌1

0.35

ments, 0.3 푋2 Equations, 0.25 Constraints Analysis 0.2 0.15 푌 0.1 Tools, etc. 20.05 0

푋 -0.05 3 -1 -0.5 0 0.5 1 1.5 2 2.5 3

QOI Design Feedback Variance, Complexity, and Risk Uncertainty Feedback

Robustness and Reliability Resource Allocation

Figure 1-2: Schematic of the proposed system design process lections of interrelated elements that interact with one another to achieve a common purpose [5]. A system can be a physical entity, such as a vehicle or consumer product, or an abstract concept, such as a software program or procedure. Whatever its form, a system must satisfy a set of targets in order to be deemed successful. These targets typically refer to functional requirements that dictate the system’s performance, but may also include additional constraints that stipulate conditions such as budget or time.

From the system targets, a set of design parameters can be enumerated which are used to characterize the system. Design parameters are variables that relate the requirements in the functional domain to aspects of the design in the physical domain that can be manipulated to satisfy those requirements [6]. They can include quantities that deﬁne the system itself, such as individual component dimensions, as well as data describing the procedures and other elements by which the system can be manufactured, operated, and maintained [7]. In this thesis, we assume that all design parameters within a system are independent, and denote them using the m×1

22 T th vector x = [x1, x2, . . . , xm] , where xi is the i design parameter among a total of m design parameters.

In addition to the design parameters, there are also quantities that are used to characterize aspects of the system’s performance that are of interest to the designer. We refer to them generically as quantities of interest (QOI), and represent them using T th the n × 1 vector y = [y1, y2, . . . , yn] , where yj is the j QOI out of a total of n. Quantities of interest are typically evaluated indirectly as functions of the design parameters using available models and analysis tools. For example, in aerospace engineering, relevant QOI can include the gross weight of an aircraft, which can be computed from design parameters regarding vehicle geometry and material selection, or the lift-to-drag ratio, which can be estimated from the external geometry using results from computational ﬂuid dynamics and wind tunnel testing. For simplicity, we will assume that there already exist models and tools (may be black-box) with which to evaluate the QOI of a system from the design parameters. We express this relationship as: y = g(x), (1.1)

where g : Rm → Rn denotes the mapping from design parameters to QOI. For the jth

QOI yj, this mapping is given by:

yj = gj(x), (1.2)

m 1 where gj : R → R , and j = 1, 2, . . . , n. With regard to analysis models and tools used to aid system design, we will also use the terms factor and output. Consistent with the deﬁnitions set forth in Ref. [8], a factor refers to an external input to a model, whereas output is a model result of interest. While these terms are similar to design parameter and quantity of interest in the context of a system, they are not equivalent. A factor can refer to any model input, and need not correspond to a parameter that can be manipulated by the designer. Similarly, the outputs of a model are not necessarily limited to the QOI of a system.

23 1.2.2 Characterizing Uncertainty

For most realistic engineering systems, the design process is made more difficult by the presence of non-deterministic features that contribute to stochasticity in the outcome, which comprise the uncertainty associated with the QOI [9, 10]. In this thesis, uncertainty quantification refers to “the science of identifying, quantifying, and reducing uncertainties associated with models, numerical algorithms, experiments, and predicted outcomes or quantities of interest” [11]. We focus specifically on the development of methods to quantify uncertainty during the intermediate stages of system design, where requirements, design parameters, and QOI have already been defined, but no feasible design has yet been realized. Uncertainty pertaining to the formulation of functional requirements, development of analysis tools, manufacturing, deployment, maintenance, and organizational structure will not be discussed. Uncertainty can come in many flavors; for example, it can be broadly categorized as epistemic, owing to insufficient or imperfect knowledge, or aleatory, which arises from natural randomness and is therefore irreducible [10–13]. Furthermore, there are several types of uncertainty associated with the use of computer-based models in system design [14]:

• Parameter uncertainty, resulting from not knowing the true values of the inputs to a model;

• Parametric variability, relating to unspeciﬁed conditions in the model inputs or parameters;

• Residual variability, due to the inherently unpredictable or stochastic nature of the system;

• Observation error, referring to uncertainty associated with actual observations or measurements;

• Model inadequacy, owing to the insuﬃciency of any model to exactly predict reality;

• Code uncertainty, arising from not knowing the output of a model given a particular set of inputs until the code is run.

24 The framework presented in this thesis is intended to be general enough to describe any of aforementioned types of uncertainty, provided that they can be described probabilistically. However, we note that although both epistemic and aleatory sources of uncertainty can be characterized within this framework, only epistemic uncertainty can be reduced through additional research and improved knowledge. Thus, while it may be possible to study the eﬀects of aleatory sources of uncertainty, it is not appropriate to allocate resources toward their reduction. To characterize the propagation of uncertainty, we employ continuous random variables to represent design parameters2 and quantities of interest [10, 18, 19], and model the time evolution of design as a stochastic process whose outcome is governed by some probability distribution. We simulate the behavior of the system using Monte Carlo (MC) sampling, which generates numerous QOI estimates from which 3 T a probability density can be estimated. We use X = [X1,X2,...,Xm] and Y = T [Y1,Y2,...,Yn] to denote vectors of random variables that correspond to each entry in x and y, respectively. The probability densities associated with Xi and Yj are given by fXi (xi) and fYj (yj), respectively. In general, these densities can be of arbitrary form, and closed-form expressions may not be available. The stochastic representations of (1.1) and (1.2) can be written as:

Y = g(X), (1.3)

Yj = gj(X). (1.4)

Once the QOI have been estimated, we use selected metrics to quantitatively describe their levels of uncertainty. This information is compared to the system’s

2The process of assigning design parameters to speciﬁc probability distributions will not be discussed in this thesis. Instead, the reader is referred to Refs. [15–17], which detail several methods addressing this topic. They include density estimation techniques to leverage historical data, elicitation of expert opinion to supplement empirical evidence, and calibration and aggregation procedures to combine information from multiple sources. 3Monte Carlo simulation is computationally expensive, and may be intractable for some systems. In practice, computational expense can be reduced by using quasi-random sampling techniques [20, 21], reducing the dimensionality of the input space [22–24], or employing less-expensive surrogate models to approximate system response [25–29]. In this thesis, we will assume that forward simulation using MC sampling is computationally feasible.

25 uncertainty targets through the uncertainty feedback loop. We also perform sensitivity analysis to identify the key drivers of uncertainty, enumerate uncertainty reduction strategies, and evaluate tradeoﬀs in the context of resource allocation decisions for uncertainty mitigation, thus enhancing system robustness and/or reliability.

1.2.3 Some Common Probability Distributions

Throughout this thesis, we will focus on four particular types of probability distributions for characterizing uncertainty in the design parameters: uniform, triangular,

Gaussian, and bimodal with uniform peaks, which are shown in Figure 1-3.

푥 푥

푋 푋

푓 푓

푎 푏 푎 푐 푏 푥 푥 (a) Uniform distribution with lower bound (b) Triangular distribution with lower

a and upper bound b bound a, upper bound b, and mode c

푥

푋

푋 푓 푓 푓퐿 푓푅 휎

휇 푎퐿 푏퐿 푎푅 푏푅 푥 푥 (c) Gaussian distribution with mean µ and (d) Bimodal distribution composed of standard deviation σ uniform peaks with bounds aL, bL, aR, and bR, and peak heights fL and fR

Figure 1-3: Examples of uniform, triangular, Gaussian, and bimodal uniform probability densities

26 A uniform distribution is speciﬁed by two parameters, a and b, which correspond to the lower and upper bound of permissible values, respectively. All values on the interval [a, b] are equally likely, and therefore a uniform distribution is often assumed when no knowledge exists about a parameter except for its limits. The probability density of a uniform random variable X is given by:

 1  if a ≤ x ≤ b, b − a fX (x) = (1.5)  0 otherwise.

Similar to the uniform distribution, a triangular distribution is also parameterized by a lower bound a and an upper bound b. There is also an additional parameter, c, which represents the mode. A triangular distribution is appropriate when information exists about the bounds of a particular quantity, as well as the most likely value. The corresponding probability density is:

 2(x − a)  if a ≤ x ≤ c, (b − a)(c − a)   2(b − x) fX (x) = if c < x ≤ b, (1.6) (b − a)(b − c)   0 otherwise.

Unlike uniform and triangular distributions, a Gaussian distribution does not have ﬁnite bounds. It is often used when there is strong conﬁdence in the most likely value [17]. A Gaussian distribution is parameterized by its mean µ and standard deviation σ. Its probability density is nonzero everywhere along the real line:

1 (x − µ)2 fX (x) = √ exp − . (1.7) 2πσ2 2σ2

In practice, it is often unrealistic to work with random variables with inﬁnite support. Instead, it may be necessary to approximate a Gaussian distribution with a truncated form [30]. Bimodal distributions are used to characterize quantities consisting of two distinct

27 sets of features or behaviors. They can arise in a variety of science and engineering contexts: for example, in materials science, to characterize the size and properties of polymer particles [31–33]; in medicine, to explain disease epidemiology within and between populations [34, 35]; in human factors, to analyze individual and group communication patterns [36, 37]; in earth science, to describe the occurrence of natural phenomena such as rainfall and earthquakes [38–41]; and in electrical engineering, to relate disparities in circuit component lifetime to distinct modes of failure [42, 43].

For simplicity, we will restrict our consideration to bimodal distributions composed of two uniform peaks, henceforth termed “bimodal uniform,” as shown in Figure 1-

3(d). We use [aL, bL] and [aR, bR] to denote the lower and upper bounds of the left

and right peaks, respectively, and fL and fR to designate the peak heights. It takes ﬁve parameters to uniquely specify a distribution of this type; therefore, given any

ﬁve of {aL, bL, fL, aR, bR, fR}, the sixth can be directly computed from the condition that:

fL(bL − aL) + fR(bR − aR) = 1. (1.8)

The probability density of a bimodal uniform distribution is given by:

 f if a ≤ x ≤ b ,  L L L  fX (x) = fR if aR ≤ x ≤ bR, (1.9)   0 otherwise.

1.3 Designing for Robustness and Reliability

Uncertainty in engineering design is closely related to the robustness and reliability of the resulting systems. In this section, we provide deﬁnitions for these terms, discuss current practices in design for robustness and reliability, as well as identify several opportunities for intellectual contribution.

28 1.3.1 Deﬁning Robustness and Reliability

According to Knoll and Vogel, robustness is “the property of systems that enables them to survive unforeseen or unusual circumstances” [44]. That is to say, to improve a system’s robustness is to make it less susceptible to exhibit unexpected behavior in the presence of a wide range of stochastic elements [45, 46]. Reliability, on the other hand, describes a system’s “probability of success in satisfying some performance criterion” [9], and is closely tied to system safety [45].4 It is also related to the deﬁnition of risk that we will introduce in Section 2.4, which poses risk and reliability as complementary quantities. Refinement: Change Std Dev Redesign: Shift Mean

푓푌 푦 0.8 0.8

푓푌 푦 0.6 0.6 Constant Shift

Constant (x)

0.4 Scaling (x) 0.4

X f f 푃 푌 < 푟

0.2 푃 푌 < 푟 0.2

0 푦 0 푦 푟 푟

-0.2 -0.2 -4 -3 (a)-2 Design-1 for0 robustness1 2 Redesign3 4 and-45 Refinement-3 (b)-2 Design-1 0 for1 reliability2 3 4 5 x x 푓푌 푦 0.8

0.6 Constant Shift &

(x) 0.4 Scaling

X f

0.2 푃 푌 < 푟

0 푦 푟

-0.2 (c)-4 Design-3 -2 for-1 robustness0 1 and2 3 4 5 reliability x

Figure 1-4: Schematics of design for robustness and reliability. Red and blue probability densities represent initial and updated estimates of fY (y), respectively. Dashed line indicates the location of the requirement r. Shaded portions denote regions of failure in which the requirement is not satisﬁed.

4Alternate deﬁnitions of reliability also ascribe an element of time. For example, reliability is sometimes described as the probability of failure within a given interval of time [47], or the expected life of a product or process, typically measured by the mean time to failure or the mean time between failures [46]. In this thesis, we will not address the temporal aspect of reliability.

29 Graphically, we illustrate the distinction between design for robustness and design for reliability in Figure 1-4, in which our estimate of a QOI Y is updated from the red probability density to the blue. The dashed line y = r represents a performance criterion below which the design is deemed unacceptable; the area of the shaded region corresponds to P (Y < r), the probability of failure.5 Figure 1-4(a) shows a constant

scaling of fY (y) with no change in the mean value, whereas Figure 1-4(b) depicts

a constant shift (horizontal translation) of fY (y) with no change in the standard deviation. Both activities have direct implications for the uncertainty associated with Y . In practice, improvements in robustness and reliability can occur concurrently, as depicted in Figure 1-4(c). In the following chapters, however, we will treat them as distinct activities in order to investigate how robustness and reliability are related to two proposed measures of uncertainty — complexity and risk — as well as how sensitivity analysis can be used to predict changes in these quantities.

1.3.2 Background and Current Practices

The origins of robust design, also known as parameter design, can be traced to post- World War II Japan, where it was pioneered by Genichi Taguchi as a way to improve design eﬃciency and stimulate industries crippled by the war [48]. The aim of the so-called “Taguchi Methods” is to reduce the sensitivity of products and processes to various noise factors, such as manufacturing variability, environmental conditions, and degradation over time [49, 50]. To achieve this goal, Taguchi applied design of experiment techniques [51, 52], in particular the orthogonal array, to systematically vary the levels of the control factors (e.g., the design parameters) and noise factors, quantify the eﬀect in the output, and compute a signal-to-noise ratio. In order to improve system robustness, the designer seeks to determine the parameter values that maximize the signal-to-noise ratio, thus minimizing the system’s sensitivity to variability. Finally, a loss function is used to estimate the expected decrement in quality due to deviation of the system response from a target value.

5The probability of failure can also be deﬁned as P (Y > r) in the case that the QOI must not exceed r, noting that P (Y > r) = 1 − P (Y < r).

30 More recently, the robust design literature has expanded to include techniques such as the combined array format [53, 54], response modeling [53–55], and adaptive one-factor-at-a-time experimentation [56, 57]. Whereas traditional Taguchi methods evaluate the inﬂuence of control factors and noise factors separately, the combined array format merges the two sets in a single experimental design, thereby reducing the number of required experiments. In response modeling, the output of the system is modeled and used as the basis for design optimization, rather than the loss function. One advantage of this approach is that it allows the designer to model the mean and variance of the output separately, and thus gain an improved understanding of underlying system behaviors [55, 58]. Finally, in adaptive one-factor-at-a-time experimentation, the designer begins with a baseline set of factor levels, then varies each factor in turn to a level that has not yet been tested while holding all other factors constant. Along the way, the designer also seeks to optimize the system response by keeping a factor at its new level if the response improves, or reverting to its previous level if the response worsens. Although this approach does not guarantee optimal factor settings, it has been shown in certain situations to be more eﬀective than traditional fractional factorial experimentation for improving design robustness [56, 57, 59, 60].

Another design philosophy used in many industrial sectors today is what is known as “Six Sigma.” First introduced by Motorola, Inc. in the mid-1980s, Six Sigma is a set of strategies and techniques aimed to minimize risk by reducing the sources of variability in a system, not just the system’s sensitivity to said variability [46, 61]. Qualitatively, to design for Six Sigma is to ensure that the mean system performance far exceeds the design requirement such that failures are extremely rare. In this approach, uncertainty in the QOI is represented using a Gaussian distribution with mean µ and standard deviation σ (or “sigma”), which is subject to a failure criterion r. Based on experimental testing, the number of sigmas that a particular design achieves is estimated as the standard score (also known as the z value or z score) of

31 r with respect to the observed µ and σ, expressed by:

r − µ z = . (1.10) σ

From (1.10), the standard Gaussian cumulative distribution function Φ(z) can be used to compute the reliability of the system with respect to the QOI. Quantitatively, Six Sigma is deﬁned by a 99.99966% probability of success, or 3.4 defects per million.6 Note that because Six Sigma seeks to maximize the probability of success through variance reduction, it has implications for both system robustness and reliability (i.e., see Figure 1-4(a)).

In the design for reliability context, a common approach to minimize risk is to ensure that a system’s nominal performance deviates from a critical requirement by a speciﬁed amount. This is typically done by applying a multiplicative factor of safety or an additive margin of safety (usually expressed as a percentage) to the requirement, thus injecting conservatism into the design [5].

Another class of well-known methods are the First- and Second-Order Reliability Methods (FORM and SORM, respectively). Originally used in civil engineering to assess the reliability of structures under combined loads, FORM linearize a system about the current design point in order to estimate the mean and standard deviation of the resulting QOI [9, 63, 64]. As in Six Sigma, all random variables are assumed to be Gaussian and uncorrelated, a z score (termed the “reliability index” in FORM) is computed from the estimates of µ and σ, and the system is assigned a probability of failure according to 1 − Φ(z). The goal of FORM is to employ various optimization techniques to modify the design in such a way as to maximize the reliability index. A similar approach is adopted in SORM, except that a second-order approximation of the system is used [65, 66].

6Confusingly, the 3.4 defects per million benchmark does not actually correspond to six standard deviations of a Gaussian distribution. In fact, a failure probability of 3.4 × 10−6 corresponds to 1 − Φ(4.5), or “4.5 sigma.” By convention, the extra 1.5 sigma shift is introduced to account for degradation in reliability over time [62]. The actual six sigma failure probability is 1 − Φ(6) = 9.9 × 10−10, or approximately 1 defect per billion.

32 1.3.3 Opportunities for Intellectual Contribution

In view of current practices in designing for robustness and reliability, this thesis proposes to make intellectual contributions in three main areas. First, many existing methods use variance as the sole measure of uncertainty. Although variance has many advantages as an uncertainty metric (see Section 2.3.1), we will show in the following chapters that it also has several limitations, and should be supplemented with additional measures such as complexity and risk in order to provide a richer description of system robustness and reliability. Second, a common theme in existing methods is the use of Gaussian distributions to represent uncertainty in a system’s design parameters and QOI. As we will demonstrate, this simplifying assumption can lead to under- or overestimates of various uncertainty measures, as well as erroneous prioritization of efforts for uncertainty mitigation. In this thesis, we will investigate the influence of distribution shape in system design as it relates to uncertainty and sensitivity estimates. Third, current methods for designing for robustness and reliability typically emphasize variance reduction, but do not provide specific details on how it is to be achieved. Thus, a potential opportunity lies in using uncertainty and sensitivity information to identify specific strategies to mitigate uncertainty. We propose to do this by evaluating the tradeoffs between available design options with respect to robustness and reliability, and by incorporating cost and uncertainty budgets to aid decisions regarding resource allocation.

1.4 Thesis Objectives

The primary goal of this thesis is to quantify and understand the eﬀects of uncertainty in the design of engineering systems in order to guide decisions aimed to improve system robustness and reliability. Speciﬁcally, the thesis objectives are:

1. To deﬁne and apply an information entropy-based complexity metric to estimate uncertainty in the design of engineering systems.

33 2. To develop a sensitivity analysis methodology that identiﬁes the key contribu- tors to uncertainty, evaluates tradeoﬀs between various design alternatives, and informs decisions regarding the allocation of resources for uncertainty mitigation.

3. To apply uncertainty quantiﬁcation and sensitivity analysis methods to an engineering case study to demonstrate their utility for enhancing system robustness and reliability.

1.5 Thesis Outline

This thesis is composed of six chapters. The structure and content of the remaining chapters are outlined below.

• Chapter 2 gives an overview of various metrics used to describe uncertainty, and proposes complexity and risk as two possible uncertainty measures in the context of design for robustness and reliability. Deﬁnitions are presented for each of these terms, as well as a discussion of how they can be applied to characterize uncertainty in various quantities of interest.

• Chapter 3 provides a discussion of existing sensitivity analysis methods and how they can be used to identify the key drivers of uncertainty. New local sensitivity analysis techniques are introduced, which can be used to predict changes in complexity and risk associated with a reduction in variance. In addition, a novel approach is presented that decomposes output uncertainty into contributions due to the intrinsic extent and non-Gaussianity of the input factors and their interactions.

• Chapter 4 describes how uncertainty and sensitivity information can be used in conjunction with design budgets to identify speciﬁc uncertainty mitigation strategies, visualize tradeoﬀs between available options, and guide the allocation of resources aimed at improving system robustness and reliability.

34 • Chapter 5 presents a case study in which the methods developed in this thesis are applied to assess the net cost-beneﬁt of a set of real-world aviation noise stringency options and support the design of cost-eﬀective policy implementation responses.

• Chapter 6 summarizes the key contributions of this thesis and highlights some areas of future work.

Chapter 2

Quantifying Complexity and Risk in System Design

There are many possible ways to quantify uncertainty in a system. In this chapter, we propose complexity and risk as two specific measures of uncertainty, which can be used to supplement variance and convey system robustness and reliability. Sec- tion 2.1 presents a brief overview of the literature on complexity in system design, with a particular focus on Shannon’s differential entropy. Section 2.2 proposes a new metric for complexity based on Campbell’s exponential entropy, presents the associated qualitative and quantitative definitions, and provides some possible interpretations of complexity. Section 2.3 discusses how this complexity metric relates to several other quantities that can be used to characterize uncertainty. An analogous set of definitions is also proposed for quantifying risk, which is presented in Section 2.4.

2.1 Background and Literature Review

Complexity in system design is an elusive concept for which many deﬁnitions have been proposed. Early work in the ﬁeld of complexity science by Warren Weaver posited complexity as the nebulous middle ground between order and chaos, a region in which problems require “dealing simultaneously with a sizeable number of factors

37 which are interrelated into an organic whole” [67]. Another interpretation of this idea considers a set of “phase transitions” during which the fundamental features of a system undergo drastic changes [68]. As an illustrative example, consider the phase transitions of water [69]. On one end of the spectrum, water is frozen into a simple lattice of molecules whose structure and behavior are straightforward to understand. At the other extreme, water in gaseous form consists of millions of molecules vibrating at random, and the study of such a system requires methods of statistical mechanics or probability theory [70]. In between the two lies the complex liquid state, wherein water molecules behave in a manner neither orderly nor chaotic, but at once enigmatic and mesmerizing, which has captured the imagination of fluid dynamicists past and present. Although the above example makes the idea of complexity relatable to a large audience, the debate over its definition still persists. When asked for the meaning of complexity, Seth Lloyd once replied, “I can’t define it for you, but I know it when I see it” [69]. While complexity may be in the eye of the beholder, many researchers agree that there are several properties that complex systems tend to share [71–73]:

• They consist of many parts.

• There are many interactions among the parts.

• The parts in combination produce synergistic eﬀects that are not easily predicted and may often be novel, unexpected, even surprising.

• They are diﬃcult to model and to understand.

2.1.1 Quantitative Complexity Metrics

In addition to qualitative descriptions of complexity, there have also been many at- tempts to explain complexity using quantitative measures. These metrics can be clas- siﬁed into three general categories: structure-based, process-based, and information- based. Structure-based metrics quantify the complexity associated with the physical representation of a system [74]. They typically involve counting strategies: in software

38 engineering, the source lines of code can be used to describe a computer program [75]; in mechanical design, analogous measures include the number of parts [76], functions [77], or core technologies [78] embodied in a product. Although appealing in their simplicity, these counting metrics may be susceptible to diﬀerent interpretations of what constitutes a distinct component — depending on the level of abstraction, a component may be as high-level as an entire subsystem, or as basic as the nuts and bolts holding it together. More sophisticated structure-based metrics also attempt to address the issue of component interactions. For example, McCabe proposed the idea of cyclomatic complexity in software engineering, which uses graph theory to determine the number of control paths through a module [79]. Numerous others have also recommended approaches to estimate system complexity by characterizing the number, extent, and nature of component interactions, which govern the interconnectivity and solvability of a system [73, 80–82]. Overall, structure-based complexity metrics are usually easy to understand and to implement, but they may not be meaningful except in the later stages of design, after most design decision have been made, and the system is well-characterized [83].

A second class of complexity metrics quantiﬁes system uncertainty in terms of processes required to realize the system. Metrics in this category can measure, for example, the number of basic operations required to solve a problem (computational complexity) [84, 85], the compactness of an algorithm needed to specify a message (algorithmic complexity or Kolmogorov complexity) [86–88], or the amount of eﬀort necessary to design, modify, manufacture, or assemble a product [6, 74, 83, 89].

A third possible interpretation of complexity is related to a system’s information content, often described using the notion of entropy. Like complexity, entropy is also an abstruse concept.1 Early work by thermodynamicists Carnot and Clausius sought to use entropy to describe a system’s tendency towards energy dissipation; later, Gibbs and Boltzmann formalized this notion by giving it a statistical basis, conceptualizing entropy as a measure of the uncertainty in the macroscopic state of

1The Hungarian-American mathematician John von Neumann once famously said “nobody knows what entropy really is, so in a debate you will always have the advantage” [90].

39 a system associated with the microstates that are accessible during the course of the system’s thermal ﬂuctuations. For a system with probability p(l) of accessing microstate l, the Gibbs entropy S is deﬁned as:

X (l) (l) S = −kB p log p , (2.1) l

−23 where kB = 1.38065 × 10 J/K is the Boltzmann constant. From its origins in thermodynamics and statistical mechanics, entropy has since been adapted to describe randomness and disorder in many other contexts, including information theory [91], economics [92], and sociology [93–95]. Most relevant to this thesis is the concept of information entropy, which was originally proposed by Claude Shannon to study lossless compression schemes for communication systems [91]. It is discussed in more detail in the following section; in Section 2.2.1, we will introduce a complexity metric that is based on information entropy.

2.1.2 Diﬀerential Entropy

Shannon’s information entropy is a well-known measure of uncertainty, and represents the amount of information required on average to describe a random variable. For a discrete random variable Y that can assume any of k values y(1), y(2), . . . , y(k), the

Shannon entropy is deﬁned in terms of the probability mass function pY (y):

k X (l) (l) H(Y ) = − pY (y ) log pY (y ). (2.2) l

The quantity H(Y ) is always nonnegative. In the limiting case, if Y is deter- ∗ ∗ ministic with a value of y , then pY (y) = 1 for y = y , and zero elsewhere, which results in H(Y ) = 0. The units of H(Y ) depend on the base of the logarithm used in (2.2); base 2 and base e are the most common, resulting in units of “bits” or “nats,” respectively. In this thesis, we will deal exclusively with the natural logarithm. Com- paring (2.1) and (2.2), it is easy to see that information entropy has an intuitive and appealing analogy to thermodynamic entropy as a measure of a system’s tendency

40 toward disorder [96].

In the design of complex systems, design parameters and quantities of interest typically can vary over a range of permissible values. In order to avoid restricting these quantities to take only discrete values, we model them as continuous random variables with associated probability densities. For a continuous random variable Y , the analogy to Shannon entropy is diﬀerential entropy, which is deﬁned in terms of fY (y) over the support set Y:

Z h(Y ) = − fY (y) log fY (y) dy. (2.3) Y

Unlike Shannon entropy, differential entropy can take on values anywhere along the real line. For example, let Y be a uniform random variable on the interval [a, b]. −1 Using (2.3) with fY (y) = (b − a) for y ∈ [a, b], the differential entropy is computed to be h(Y ) = log(b − a). Clearly, h(Y ) > 0 if b − a > 1, h(Y ) = 0 if b − a = 1, and h(Y ) → −∞ as b − a → 0. This result comes about because whereas Shannon entropy is an absolute measure of randomness in a thermodynamic sense, differential entropy measures randomness relative to a reference coordinate system [88, 91].

2.2 A Proposed Deﬁnition of Complexity

In this thesis, we use complexity as a measure of uncertainty in a system. Qual- itatively, we deﬁne complexity as the potential of a system to exhibit unexpected behavior in the quantities of interest, whether detrimental or not [97]. This deﬁnition is consistent with many of the theoretical formulations of complexity discussed in Sec- tion 2.1, such as the notions of emergent behavior, nonlinear component interactions, and information content. It is also in line with the view that reducing complexity translates to minimizing a system’s information content and maximizing its likelihood of satisfying all requirements [74, 98].

41 2.2.1 Exponential Entropy as a Complexity Metric

Quantitatively, we use exponential entropy as our complexity metric. Exponential entropy was ﬁrst proposed by L. Lorne Campbell to measure the extent of a probability distribution [99]. For a continuous random variable Y , the simplest measure of extent is the range, ν(Y ), deﬁned as the length of the interval on which the probability density is nonzero:

Z ν(Y ) = dy. (2.4) Y

In the case where Y is a uniform random variable on the interval [a, b], ν(Y ) = b−a as expected. One disadvantage of range, however, is that it gives equal weight to all possible values of y ∈ Y, and consequently can be inﬁnite, as is the case for probability distributions such as Gaussian or exponential. In these cases, range is no longer a useful measure of uncertainty; instead, it is desirable to formulate a measure of “intrinsic extent.” Campbell put forward such a quantity in exponential entropy, which is closely related to Shannon’s diﬀerential entropy [99]. The derivation of exponential entropy is given below.2

First, consider Mt(Y ), the generalized mean of order t of fY (y) over Y [100]:

 Z t !1/t  1  fY (y) dy if t 6= 0,  Y fY (y) Mt(Y ) = (2.5)  Z  1 exp log fY (y) dy if t = 0. Y fY (y)

From (2.5), it is easy to see that ν(Y ) = M1(Y ); that is, range is the generalized

mean of order one. Furthermore, when t = 0, M0(Y ) is the geometric mean of fY (y) over Y [101].

For arbitrary orders s and t, where s < t, the relationship between Ms(Y ) and

2 Instead of working with fY (y) and Y, Campbell’s derivation uses the Radon-Nikodym derivative of a probability space (Ω, A,P ), where Ω is the sample space, A is a σ-algebra, and P is a probability measure. Thus, it is more general than our presentation, which applies to probability densities deﬁned over real numbers, in that it also extends to probability measures deﬁned over arbitrary sets.

42 Mt(Y ) can be described by the generalized mean inequality [100]:

Ms(Y ) ≤ Mt(Y ), (2.6) where equality holds if and only if Y is a uniform random variable.3 Setting s = 0 and t = 1, we can rewrite (2.6) to obtain a general relationship between M0(Y ) and ν(Y ):

M0(Y ) ≤ M1(Y ), (2.7)

M0(Y ) ≤ ν(Y ). (2.8)

Campbell terms the quantity M0(Y ) the intrinsic extent of Y , which is always less than or equal to the range ν(Y ) (i.e., the “true” extent). Another interpretation of M0(Y ) is as “the equivalent side length of the smallest set that contains most of the probability” [88]. That is to say, whereas ν(Y ) gives the length of the interval

(potentially inﬁnite) that contains all possible values y for which Y = y, M0(Y ) represents the (ﬁnite) extent of Y once the various values of y have been weighted by their probability of occurrence. In the case where Y is a uniform random variable,

M0(Y ) and ν(Y ) are equivalent.

In addition, M0(Y ) is also closely related to Shannon’s diﬀerential entropy h(Y ).

To see this, consider the natural logarithm of M0(Y ):

Z 1 log M0(Y ) = log exp log fY (y) dy (2.9) Y fY (y) Z 1 = fY (y) log dy (2.10) Y fY (y) Z = − fY (y) log fY (y) dy (2.11) Y = h(Y ). (2.12)

From (2.12), it follows that M0(Y ) is the exponential of the diﬀerential entropy of Y ,

3 If Y is a uniform random variable, Mt(Y ) = ν(Y ) = b − a for all t.

43 hence the term “exponential entropy”:

M0(Y ) = exp[h(Y )]. (2.13)

Using exponential entropy as our metric for complexity, a QOI represented by Y thus has a complexity given by: C(Y ) = exp[h(Y )]. (2.14)

2.2.2 Interpreting Complexity

As exponential entropy is a measure of intrinsic extent, it has the same units as the original random variable. We note that it is also consistent with our proposed qualitative definition of complexity, as well as fits with the notion of robustness introduced in Section 1.3. In the limiting case, a deterministic system devoid of any potential to exhibit unexpected behavior in the QOI has no complexity, and thus its exponential entropy is zero. From (2.8), (2.13), and (2.14), we have that C(Y ) ≤ ν(Y ), with equality in the case where Y is a uniform random variable. This relationship implies that of all probability distributions of a given range, the uniform distribution is the one with the maximum complexity. This result is consistent with the principle of maximum entropy, which states that if only partial information is available about a system, the probability distribution which best represents the current state of knowledge is the one with largest information entropy [102, 103]. More specifically, if the only knowledge about a system is the range [a, b] of permissible values, the largest differential entropy (and hence the greatest exponential entropy) is achieved by assuming a uniform random variable on [a, b] [104]. Another interpretation of exponential entropy is as the volume of the support set Y, which is always nonnegative [88]. For an arbitrary continuous random variable Y , the complexity of the associated quantity of interest is equivalent to that of a uniform distribution for which C(Y ) is the range. Figure 2-1 illustrates this statement for a standard Gaussian random variable Y ∼ N (0, 1). In this case,

44 Y ~ N( = 0,  = 1) Y ~ U[0, 4.13] 0.5 0.5

2 0.4 퐶 푌 = 2휋푒휎 0.4 퐶 푌 = 4.13

0.3 0.3

(y)

Y f f 0.2 0.2  Range = 4.13 0.1 0.1

0 0 -3 -2 -1 0 1 2 3 -1 0 1 2 3 4 5 y y

Figure 2-1: Probability density of a standard Gaussian random variable (left) and a uniform random variable of equal complexity (right)

Y ~ N( = [0, 0],  = I ) Y ~ U([0, 4.13], [0, 4.13]) 2

0.2 0.2

0.15 )

) 0.15 Area = 17.08

2 , y ,

, y , 0.1

0.1 1

Y f Y 0.05

f 0.05

0 퐶 푌 = 2휋푒 푛 Σ 0 2 4 2 4 0 퐶 푌 = 17.08 2 0 2 -2 -2 퐶 푌 = 4.13 2 0 0 y y y y 2 1 2 1 Y ~ U[0, 17.08] 0.2

0.15

(y) 0.1

Y f

0.05 Range = 17.08

0 0 5 10 15 y

Figure 2-2: Probability density of a multivariate Gaussian random variable with µ = (0, 0) and Σ = I2 (top left), and uniform random variables of equal complexity in R2 and R (top right and bottom, respectively)

√ C(Y ) = 2πeσ2 = 4.13, which suggests that Y is equivalent in complexity to a uniform random variable with a range of 4.13. This interpretation also holds in higher dimensions. For example, if Y is instead a multivariate Gaussian random vari-

45 able with mean µ ∈ Rd and covariance Σ ∈ Rd×d, then C(Y ) represents the volume of a d-dimensional hypercube that captures the intrinsic extent of Y , and there is a corresponding uniform random variable in Rd with the same complexity. Alterna- tively, C(Y ) can be also be represented as the range of a uniform random variable in R. Figure 2-2 illustrates this example for a standard multivariate Gaussian random variable with d = 2.

2.2.3 Numerical Complexity Estimation

In realistic complex systems, estimates of the QOI are often generated from the outputs of numerous model evaluations, such as through Monte Carlo simulation. The resulting distribution for Y can typically be of any arbitrary form; that is, no closed-form expression is available for fY (y). This precludes computing C(Y ) through the exact evaluation of h(Y ) using (2.3), requiring instead numerical integration [105].

Assuming that fY (y) is Riemann-integrable, h(Y ) can be estimated by discretizing (l) (l) fY (y) into k bins of size ∆ , where l = 1, ..., k. Letting fY (y ) denote the probability density of Y evaluated in the lth bin, and h(Y ∆) represent the estimate of h(Y ) computed using numerical approximation, we have [88]:

k ∆ X (l) (l) (l) (l) h(Y ) = − fY (y )∆ log[fY (y )∆ ] (2.15) l=1 k (l) (l) X fY (y )∆ = − f (y(l))∆(l) log (2.16) Y ∆(l) l=1 k k X (l) (l) (l) (l) X (l) (l) (l) = − fY (y )∆ log[fY (y )∆ ] + fY (y )∆ log ∆ . (2.17) l=1 l=1

k (l) X (l) If we select bins of equal size (∆ = ∆ ∀ l), then fY (y )∆ ≈ 1, which gives: l=1

k ∆ X (l) (l) h(Y ) = − fY (y )∆ log[fY (y )∆] + log ∆. (2.18) l=1

46 Note that h(Y ∆) → h(Y ) as ∆ → 0. Figure 2-3 provides an illustration for evaluating h(Y ∆) using this discretization procedure.

푓푌 푦 푙 푓푌 푦

∆

푦 bin 푙

Figure 2-3: Numerical approximation of h(Y ) using k bins of equal size

2.3 Related Quantities

In this section, we examine the relationship between exponential entropy and several other quantities that can be used to describe uncertainty. As its connection to diﬀerential entropy and range have already been addressed in previous sections, we will focus on how exponential entropy is related to variance and entropy power. We also introduce the Kullback-Leibler divergence, which will play a key role in entropy power-based sensitivity analysis, to be discussed in Chapter 3. Table 2.1 lists the expressions for several uncertainty metrics for the four types of distributions introduced in Section 1.2.3. Table 2.2 shows how the various metrics are aﬀected by the transformations shown in Figures 1-4(a) and 1-4(b), where a QOI Y is updated to a new estimate Y 0 through scaling or shifting by a constant amount.4

4 In Figure 1-4(a), fY (y) is scaled by a constant α with no change in µY . To represent this update 0 using Y = αY requires that µY = 0. Instead, the action depicted in Figure 1-4(a) is more accurately 0 described by Y = α(Y − µY ). However, the results presented in Table 2.2 for multiplicative scaling by α hold in general, and do not restrict Y to be zero mean.

47 Uniform Triangular Gaussian Bimodal Uniform Y ∼ U[a, b] Y ∼ T (a, b, c) Y ∼ N (µY , σY ) Y ∼ BU(aL, bL, aR, bR, fL, fR) Range b − a b − a Inﬁnite (b − a ) + (b − a ) ν(Y ) L L R R

Mean a + b a + b + c fL 2 2 fR 2 2 µY (b − a ) + (b − a ) E [Y ] 2 3 2 L L 2 R R 2 2 2 2 Variance (b − a) a + b + c − ab − ac − bc 2 fL 3 3 fR 3 3 2 σY (bL − aL) + (bR − aR) − (E [Y ]) var (Y ) 12 18 3 3

2 48 Diff. Entropy 1 b − a log(2πeσY ) log(b − a) + log −fL(bL − aL) log fL − fR(bR − aR) log fR h(Y ) 2 2 2 √ Exp. Entropy (b − a) e q b − a 2πeσ2 f −fL(bL−aL)f −fR(bR−aR) C(Y ) 2 Y L R Entropy Power 2 2 −2fL(bL−aL) −2fR(bR−aR) (b − a) (b − a) 2 fL fR σY N(Y ) 2πe 8π 2πe Table 2.1: Range, mean, variance, differential entropy, exponential entropy, and entropy power for distributions that are uniform, triangular, Gaussian, or bimodal with uniform peaks Constant Scaling Constant Shift Y 0 = αY, α > 0 Y 0 = β + Y, β ∈ R Variance α2var (Y ) var (Y ) var (Y 0) Diff. Entropy h(Y ) + log α h(Y ) h(Y 0) Exp. Entropy αC(Y ) C(Y ) C(Y 0) Entropy Power α2N(Y ) N(Y ) N(Y 0)

Table 2.2: The properties of variance, diﬀerential entropy, exponential entropy, and entropy power under constant scaling or shift

2.3.1 Variance

Variance is one of the most widely used metrics for characterizing uncertainty in system design. It provides a measure of the dispersion of a random variable about its mean value [106]. For a continuous random variable Y , variance is denoted by 2 var (Y ) or σY , and can be expressed in integral form as:

Z 2 2 var (Y ) = σY = (y − µY ) fY (y) dy (2.19) ZY 2 2 = y fY (y) dy − µY , (2.20) Y where µY denotes the mean (also known as the expected value or ﬁrst moment) of Y , given by: Z E [Y ] = µY = yfY (y) dy. (2.21) Y There are several advantages to using variance for quantifying uncertainty. For most distributions, variance can easily be estimated from sample realizations of the random variable, without explicit knowledge of the probability density.5 Furthermore,

5It is important to note, however, that variance is not always well-defined. For example, the variance of a Cauchy distribution, which has heavy tails, is undefined. However, its differential

49 there is a rich body of literature in variance-based sensitivity analysis, which provides rigorous methods to apportion a system’s output variability into contributions from the inputs (see Section 3.1.1). Although variance and diﬀerential entropy can both be used to describe a system’s uncertainty or complexity, in general there is no explicit relationship between the two quantities [107, 108]. Consider the example shown in Figure 2-4: let Y be a bimodal

random variable consisting of two symmetric uniform peaks, such that fY (y) = 0.5 for −2 ≤ y ≤ −1 and 1 ≤ y ≤ 2, and zero elsewhere. We deﬁne Y G to be its corresponding “equivalent Gaussian distribution” — that is, a Gaussian random variable G 6 with the same mean and variance, parameterized by Y ∼ N (µY , σY ). Intuitively, we might expect that the complexity of Y is less than that of Y G, as the former can take on values in one of only two groups, whereas the latter has inﬁnite support. For this case, var (Y ) = var (Y G) whereas C(Y ) < C(Y G), and thus exponential entropy more accurately captures the limited range of possible outcomes in Y as compared to Y G.

0.6

0.5 f (y) Y Y Y G 0.4 var (·) 2.33 2.33 0.3

0.2 h(·) 0.69 1.84 f G(y) Y Probability Density 0.1 C(·) 2.00 6.31

0 −4 −2 0 2 4 y Figure 2-4: Comparison of variance, diﬀerential entropy, and complexity between a bimodal uniform distribution and its equivalent Gaussian distribution

Based on exponential entropy, the bimodal uniform distribution has the same complexity as a uniform random variable whose range is equal to two. This illustrates another key feature of our proposed complexity metric — unlike variance, which entropy is given by log(γ) + log(4π), where γ > 0 is the shape parameter, and thus its exponential entropy is well-deﬁned. This represents one potential advantage of our proposed complexity metric, in that it can be estimated for any probability density using the procedure outlined in Section 2.2.3. 6The notion of an equivalent Gaussian distribution will be explored in more detail in Section 3.4, where it is used to characterize the non-Gaussianity of the factors and outputs of a model.

50 measures dispersion about the mean, exponential entropy reﬂects the intrinsic extent of a random variable irrespective of the expected value.

2.3.2 Kullback-Leibler Divergence and Cross Entropy

Another quantity relevant to our discussion of complexity is the Kullback-Leibler (K-L) divergence, which is a measure of the diﬀerence between two probability distri-

butions [109]. For continuous random variables Y1 and Y2, the K-L divergence from

Y1 to Y2 is deﬁned as:

Z ∞ fY1 (y) DKL(Y1||Y2) = fY1 (y) log dy (2.22) −∞ fY2 (y) Z ∞ Z ∞

= fY1 (y) log fY1 (y) dy − fY1 (y) log fY2 (y) dy (2.23) −∞ −∞

= −h(Y1) + h(Y1,Y2). (2.24)

As shown in (2.24), the K-L divergence is equal to the sum of two terms: the negative of the diﬀerential entropy h(Y1) and the cross entropy between Y1 and Y2, deﬁned as: Z ∞

h(Y1,Y2) = − fY1 (y) log fY2 (y) dy. (2.25) −∞

Note that neither K-L divergence nor cross entropy is a true metric as they are

not symmetric quantities — that is, in general DKL(Y1||Y2) 6= DKL(Y2||Y1) and

h(Y1,Y2) 6= h(Y2,Y1). Because of this, K-L divergence is also commonly referred to as the relative entropy of one random variable with respect to another, which is always 7 non-negative. In the limiting case where Y1 = Y2, we have that h(Y1,Y2) = h(Y1)

and h(Y2,Y1) = h(Y2), and thus DKL(Y1||Y2) = DKL(Y2||Y1) = 0.

7 In (2.22), Y2 is designated as the reference random variable. If fY2 (y) = 1 over the support set Y2 (e.g., a uniform random variable with an extent of one), then DKL(Y1||Y2) equals −h(Y1). This lends some intuition about the statement that diﬀerential entropy is a measure of randomness with respect to a ﬁxed reference [88, 91].

51 2.3.3 Entropy Power

In his seminal work on information theory, Shannon proposed not only the use of diﬀerential entropy h(Y ) to quantify uncertainty in a random variable, but also introduced a derived quantity N(Y ), which he termed entropy power [91]:

exp[2h(Y )] N(Y ) = . (2.26) 2πe

Entropy power is so named because in the context of communication systems, it represents the power in a white noise signal that is limited to the same band as the original signal and having the same diﬀerential entropy. As Shannon writes, “since white noise has the maximum entropy for a given power, the entropy power of any noise is less than or equal to its actual power” [91]. Due to the 2πe normalizing constant in (2.26), entropy power also has the interpretation of the “variance of a Gaussian random variable with the same diﬀerential entropy, [which is] maximum and equal to the variance when the random variable is Gaussian” [110].8 That is to say, for an arbitrary random variable Y , N(Y ) ≤ var (Y ), with equality in the case where Y is a Gaussian random variable (see Table 2.1).

Additionally, entropy power is always non-negative, and is proportional to the square of exponential entropy. Rewriting (2.26), we obtain:

exp[h(Y )] exp[h(Y )] N(Y ) = 2πe C(Y )2 = . (2.27) 2πe

Whereas C(Y ) is a measure of intrinsic extent and has the same units as Y , N(Y ) exhibits properties of variance and has the same units as Y 2 (see Table 2.2). This provides a very appealing analogy for the relationship between standard deviation and variance to that between exponential entropy and entropy power.

8This is consistent with the principle of maximum entropy, which states that if we known nothing about a system’s behavior except for the ﬁrst and second moments, the distribution that maximizes diﬀerential entropy (and therefore entropy power) given our limited knowledge is the Gaussian [104].

52 2.4 Estimating Risk

Similar to complexity, the concept of risk in system design is also difficult to for- malize. In engineering systems, the risk associated with a particular event is often characterized by two features: the probability of the event’s occurrence (likelihood) and the severity of the outcome should the event occur (consequence) [5, 111]. In one common risk management approach, these attributes are multiplied together to generate a quantitative measure of risk associated with the event, which is then used to set design priorities accordingly [112]. In this thesis, we focus only on the first component of risk, namely, the probability of a system incurring an undesirable outcome, such as a technical failure or the exceedance of a cost constraint. Quantitatively, we compute risk as the probability of failing to satisfy a particular criterion [9, 113]. Note that in this definition, risk is the complement of reliability, which instead describes the probability of success. Suppose that a QOI is subject to the requirement that it must be greater than or equal to a specified value r. In this case, risk corresponds to the probability that the random variable Y takes on a value less than r, given by:

Z r P (Y < r) = fY (y) dy. (2.28) −∞

This notion of risk is illustrated in Figure 2-5, where P (Y < r) corresponds to the area of the shaded region in which the requirement is not met. Conversely, P (Y > r) = 1 − P (Y < r).

푓푌 푦

푃 푌 < 푟

푦 푟

Figure 2-5: Probability of failure associated with the requirement r

53 2.5 Chapter Summary

In this chapter, we introduced complexity and risk as possible measures of uncertainty in a system with respect to a quantity of interest, and provided deﬁnitions for these terms. In particular, the use of exponential entropy as a complexity metric was described in detail, and placed in the context of information-based measures of uncertainty. In the following chapter, we will explore the topic of sensitivity analysis, which is used to explain how the various factors of a model contribute to uncertainty in the output, and how this information can be leveraged to reduce uncertainty in the context of improving robustness and reliability.

54 Chapter 3

Sensitivity Analysis

The focus of this chapter is sensitivity analysis in the context of engineering system design. Within the framework of Figure 1-2, sensitivity analysis takes place after we have quantified uncertainty using metrics such as variance, complexity, and risk. Sensitivity analysis allows us to better understand the effects of uncertainty in a system in order to make well-informed decisions aimed at uncertainty reduction. For example, it can be used to study how “variation in the output of a model ... can be apportioned, qualitatively or quantitatively, to different sources of variation, and how the given model depends upon the information fed into it” [114]. This information can help to answer questions such as “What are the key factors that contribute to variability in model outputs?” and “On which factors should research aimed at reducing output variability focus?” [8]. Section 3.1 provides a brief overview of the sensitivity analysis literature. New contributions of this thesis are presented in Sections 3.2 through 3.4. Specifically, Section 3.2 extends existing variance-based sensitivity analysis methods to permit changes to distribution family and examine the effect of input distribution shape on output variance. Section 3.3 details how to predict local changes in complexity and risk due to perturbations in mean and standard deviation, which have direct implications for system robustness and reliability. Section 3.4 describes a sensitivity analysis approach based on entropy power, in which uncertainty is decomposed into contributions due to the intrinsic extent and non-Gaussianity of each factor. These

55 sections set the stage for Chapter 4, which explores how sensitivity information can be combined with design budgets to guide decisions regarding the allocation of resources for uncertainty mitigation.

3.1 Background and Literature Review

There are several key types of sensitivity analysis: factor screening, which identifies the influential factors in a particular system; global sensitivity analysis (GSA), which apportions the uncertainty in a model output over the entire response range into contributions from the input factors; distributional sensitivity analysis (DSA), which extends GSA to consider scenarios in which only a portion of a factor’s variance is reduced; and regional sensitivity analysis, which is similar to GSA and DSA but focuses instead on a partial response region — for example, the tail region of a distribution in which some design requirement is not satisfied. The sections below discuss several methods that are most relevant to the current thesis: variance-based GSA and DSA, and relative entropy-based sensitivity analysis.

3.1.1 Variance-Based Global Sensitivity Analysis

Global sensitivity analysis uses variance as a measure of uncertainty, and seeks to apportion a model’s output variance into contributions from each of the factors, as well as their interactions. This notion is illustrated in Figure 3-1.

Figure 3-1: Apportionment of output variance in GSA [8, Figure 3-1]

There are two main methods to perform GSA: the Fourier amplitude sensitivity test (FAST), which uses Fourier series expansions of the output function to repre-

56 sent conditional variances [115–117], and the Sobol’ method, which utilizes Monte Carlo simulation [118–121]. The Sobol’ method is based on high-dimensional model representation (HDMR), which decomposes a function g(X) into the following sum:

X X g(X) = g0 + gi(Xi) + gij(Xi,Xj) + ... + g12...m(X1,X2,...,Xm), (3.1) i i

where g0 is a constant, gi(Xi) is a function of Xi only, gij(Xi,Xj) is a function of Xi

and Xj only, etc. Although (3.1) is not a unique representation of g(X), it can be made unique by enforcing the following constraints:

1 Z k = i1, . . . , is, gi1,...,is (Xi1 ,...,Xis ) dxk = 0, ∀ (3.2) 0 s = 1, . . . , m.

In the above equation, the indices i1, . . . , is represent all sets of s integers that satisfy

1 ≤ i1 < . . . < is ≤ m. That is, for s = 1, the constraint given by (3.2) applies to all sub-functions gi(Xi) in (3.1); for s = 2, the constraint applies to all sub-functions gij(Xi,Xj) with i < j in (3.1), etc. Furthermore, in (3.2) we have deﬁned all factors

Xi1 ,...,Xis on the interval [0, 1]; this is merely for simplicity of presentation, and not a requirement of the Sobol’ method. With the sub-functions gi1,...,is (Xi1 ,...,Xis ) now uniquely speciﬁed, we refer to (3.1) as the analysis of variances high-dimensional model representation (ANOVA-HDMR) of g(X).

In the ANOVA-HDMR, all sub-functions gi1,...,is (Xi1 ,...,Xis ) are orthogonal and zero-mean, and g0 equals the mean value of g(X). Assuming that g(X) and therefore all comprising sub-functions are square-integrable, the variance of g(X) is given by:

Z 2 2 D = g(X) dx − g0, (3.3)

and the partial variances of gi1,...,is (Xi1 ,...,Xis ) are deﬁned as:

Z 2 Di1,...,is = gi1,...,is (Xi1 ,...,Xis ) dxi1 ... dxis . (3.4)

57 Squaring then integrating both sides of (3.1), we obtain:

X X D = Di + Dij + ... + D12...m, (3.5) i i

which states that the variance of the original function equals the sum of the variances of the sub-functions, as shown in Figure 3-1. We will return to this notion of uncertainty apportionment in Section 3.4.1, where the ANOVA-HDMR is used in deriving the entropy power decomposition.

In the Sobol’ method, global sensitivity indices are deﬁned as:

D S = i1,...,is , s = 1, . . . , m. (3.6) i1,...,is D

By definition, the sum of all sensitivity indices for a given function is one. Two particularly important types of sensitivity indices are the main effect sensitivity index (MSI) and the total effect sensitivity index (TSI). For a system modeled by (1.3), the th MSI of the i factor, denoted by Si, represents the expected relative reduction in the variance of Y if the true value of Xi is learned (i.e., the variance of Xi is reduced to zero): var (Y ) − [var (Y |X )] S = E i . (3.7) i var (Y )

Alternatively, Si can be expressed as the ratio of the variance of gi(Xi) to the variance of g(X): D S = i . (3.8) i D

Whereas the MSI signiﬁes the contribution to output variance due to Xi alone,

the total eﬀect sensitivity index ST i denotes the contribution to output variance due to Xi and its interactions with all other factors. Using the notation X∼i to denote all factors except Xi, ST i represents the percent of output variance that remains unexplained after the true values of all factors except Xi have been learned:

var (Y ) − [var (Y |X )] S = E ∼i . (3.9) T i var (Y )

58 Equivalently, ST i equals the sum of all global sensitivity indices of the form speciﬁed by (3.6) that contain the subscript i.

Pm Pm For a particular model, i=1 Si ≤ 1, whereas i=1 ST i ≥ 1; deviation from unity reflects the magnitude of the interaction effects. The various factors of a model can be ranked according to their MSI or TSI in a factor prioritization or factor fixing setting [122]. The goal of factor prioritization is to determine the factors that, once their true values are known, would result in the greatest expected reduction in output variability; these correspond to the factors with the largest values of Si. In factor fixing, the objective is to identify non-influential factors that can be fixed at a given value without substantially affecting the output variance; in this case, the least significant factors are those with the smallest values of ST i.

3.1.2 Vary-All-But-One Analysis

A related concept to Sobol’-based GSA is vary-all-but-one analysis (VABO), in which two Monte Carlo simulations are conducted to determine a factor’s contribution to output variability [123, 124]. In the first MC simulation, all factors are sampled from their nominal distributions, whereas in the second, one factor is fixed at a particular point on its domain while the other factors are allowed to vary. The observed difference in output variance between the two simulations is then attributed to the contribution of the fixed factor to output variability.

While the VABO method is simple to implement, its main limitation is that there is no clear guidance as to where each factor ought to be ﬁxed on its domain; depending on the chosen location, the resulting apportionment of variance among the factors can change [122]. By contrast, GSA circumvents this problem by accounting for all possible values of each factor within its domain, such that the resulting sensitivity index represents the expected reduction in variance, where the expectation is taken over the distribution of values at which the factor can be ﬁxed.

59 3.1.3 Distributional Sensitivity Analysis

Building on global sensitivity analysis methods, distributional sensitivity analysis (DSA) can also be applied to parameters whose uncertainty is epistemic in nature. The DSA procedure was developed by Allaire and Willcox in order to address one of the inherent limitations of GSA [8, 125]: the assumption that all epistemic uncertainty associated with a particular factor may be reduced to zero through further research and improved knowledge. This generalization is optimistic, and can in fact lead to inappropriate allocations of resources. Distributional sensitivity analysis, on the other hand, treats the portion of a factor’s variance that can be reduced as a random variable, and therefore may be more appropriate than GSA for the prioritization of eﬀorts aimed at uncertainty reduction, as it could convey for which input(s) directed research will yield the greatest return.

o Letting Xi be the random variable corresponding to the original distribution for 0 factor i, and Xi be the random variable whose distribution represents the uncertainty in that factor after some design modiﬁcation, the ratio of remaining (irreducible) variance to initial variance for factor i is deﬁned by the quantity δ:

0 var (Xi) δ = o . (3.10) var (Xi )

When we do not know the true value of δ, we treat it as a uniform random variable on the interval [0, 1]. Whereas GSA computes for each factor Xi the main effect sensitivity index Si, the analogous quantity in DSA is adjSi(δ), the adjusted main effect sensitivity index of Xi given that it is known that only 100(1 − δ)% of its variance can be reduced.1 Additionally, the average adjusted main effect sensitivity index, or AASi, is the expected value of adjSi(δ) over all δ ∈ [0, 1]. Another advantage of DSA is that it does not require additional MC simulations; rather, it can be implemented by reusing samples from a previous Sobol’-based GSA. Finally, the current DSA methodology assumes that a design activity that reduces the variance of Xi by 100(1−δ)% does not change the distribution family to which Xi

1The main eﬀect sensitivity indices computed from GSA correspond to the case where δ = 0.

60 belongs. That is to say, if X is initially a Gaussian random variable with variance σ2 , i Xi its distribution remains Gaussian after the speciﬁed activity, with an updated variance of δσ2 . In Section 3.2, we extend the existing DSA methodology to permit changes Xi to a factor’s distribution type. This contribution will complement the discussion in Section 3.4, which demonstrates that a distribution’s shape can play a signiﬁcant role in uncertainty estimation.

3.1.4 Relative Entropy-Based Sensitivity Analysis

In addition to variance-based methods, there has also been work done to develop methods for global and regional sensitivity analysis using information entropy as a measure of uncertainty. One method uses the Kullback-Leibler divergence, or relative entropy, to quantify the distance between two probability distributions [126].

Suppose that Y o represents the original estimate of a system’s quantity of interest, 0 and Y the modified result once some factor Xi has been fixed at a particular value (e.g., its mean), as in a VABO approach. In relative entropy-based sensitivity analysis, (2.22) is used to compute the K-L divergence between the two QOI estimates, which serves to quantify the impact of the factor that has been fixed. That is to say, the o 0 larger the value of DKL(Y ||Y ), the more substantial the contribution of factor i to uncertainty in the QOI.

This approach can be used to perform both global and regional sensitivity analysis by simply adjusting the limits of integration, and is suitable for distributions where variance is not a good measure of uncertainty (e.g., bimodal or skewed distributions). However, as in the VABO approach, the results of relative entropy-based sensitivity analysis vary depending on where a particular factor is ﬁxed on its domain. Further- more, the K-L divergence is not normalized and does not have physical meaning — it can be used to rank factors, but does not have a proportion interpretation such as shown in Figure 3-1 for variance-based sensitivity indices.

61 3.2 Extending Distributional Sensitivity Analysis to Incorporate Changes in Distribution Family

In the existing distributional sensitivity analysis methodology, it is assumed that when a design activity is performed that reduces the variance of factor X by 100(1 − δ)%, only the parameters of X’s distribution are altered, not the type of distribution itself. If the activity were to also change the distribution family to which X belongs (e.g., from Gaussian to uniform), any additional fluctuations expected in the variance of the QOI due to the switch would not be reflected in the computed adjusted main effect sensitivity indices. Of this, Allaire writes:

“[If] the distribution family of a given factor was expected to change through further research (e.g. from an original uniform distribution to a distribution in the triangular family), then reasonable distributions from the new family, given that the original distribution was from another family, could be deﬁned” [8, pp. 60].

Here, we extend DSA to permit changes in distribution family by defining reasonable new factor distributions, which represent the result of further research or additional knowledge about a factor [8]. These new distributions reflect a 100(1 − δ)% reduction in the factor’s variance; however, for each value of δ ∈ [0, 1], there is usually no unique distribution, and thus we average over k possible reasonable distributions to estimate the adjusted main effect sensitivity index.2 As a note on terminology, we will use Xo to denote the original random variable for some factor, and X0 to represent the updated random variable such that var (X0) = δvar (Xo), as in (3.10). Allaire’s DSA methodology uses acceptance/rejection (AR) sampling to generate samples from a desired distribution by sampling from a different distribution. In the current work, we also employ AR sampling to obtain samples of X0 (the desired distribution) from samples of Xo; in order to do this, there must

2 To illustrate this point, Allaire provides the example of a factor X that is uniformly distributed√ on the√ interval [0,√1]. For δ√= 0.5, there are inﬁnitely many new distributions (e.g., U[0, 2/2], U[1 − 2/2, 1], U[ 2/4, 1 − 2/4], etc.) for which the variance is 50% of the original variance.

62 exist a constant κ with the property that κfXo (x) ≥ fX0 (x) for all x [105]. The AR sampling procedure is outlined below; the steps may be repeated until suﬃciently many samples of X0 have been obtained.

Acceptance/rejection method 1. Draw a sample, xo, from Xo.

2. Draw a sample, u, from a uniform random variable on [0, 1].

o fX0 (x ) 0 o 3. If o ≥ κu, let x = x . fXo (x ) 4. Otherwise, discard xo and return to Step 1.

In Sections 3.2.1 through 3.2.3, we propose procedures to select reasonable new distributions and perform AR sampling for cases where Xo and X0 are uniform, triangular, or Gaussian. For these cases, the analytical expressions for δ are listed in Table 3.1. Note that the proposed procedures represent but one possible set of methods to generate reasonable distributions for X0, and are not intended to be deﬁnitive. Accordingly, the distributions obtained using the procedures are but a subset of possible manifestations of X0, and thus should not be taken to be exhaustive. In Appendix A, we will examine the corresponding cases in which X0 is a bimodal uniform random variable.

63 X0 Uniform Triangular Gaussian Xo U[a0, b0] T (a0, b0, c0) N (µ0, σ0)

0 0 2 2 02 02 02 0 0 0 0 0 0 02 Uniform b − a 3 (a + b + c − a b − a c − b c ) 12σ U[ao, bo] bo − ao (bo − ao)2 (bo − ao)2

3 0 0 2 02 02 02 0 0 0 0 0 0 02 Triangular 2 (b − a ) a + b + c − a b − a c − b c 18σ o o o 2 2 2 T [a , b , c ] ao2 + bo + co2 − aobo − aoco − boco ao2 + bo + co2 − aobo − aoco − boco ao2 + bo + co2 − aobo − aoco − boco 64 Gaussian (b0 − a0)2 a02 + b02 + c02 − a0b0 − a0c0 − b0c0 σ02 N [µo, σo] 12σo2 18σo2 σo2

var (X0) Table 3.1: Ratio of δ = var (Xo) for changes in factor distribution between the uniform, triangular, and Gaussian families. Rows represent the initial distribution for Xo and columns correspond to the updated distribution for X0. 3.2.1 Cases with No Change in Distribution Family

First, we focus on cases where Xo and X0 belong to the same family of distributions (i.e., the diagonal entries of Table 3.1). In these cases, the extent of X0 is always less than or equal to that of Xo, and thus AR sampling works well for generating samples of X0 from samples of Xo. The procedures provided below are adapted from Allaire [8]. Examples of reasonable distributions generated using the procedures are provided in Figure 3-2 for select values of δ and k.

3.5 7

3 6

2.5 5

2 4

1.5 3

Probability density 1 2

0.5 1

0 0 −0.2 0 0.2 0.4 0.6 0.8 1 1.2 0 0.2 0.4 0.6 0.8 1 x (a) Uniform to uniform: Xo ∼ U[0, 1], δ = 0.1, (b) Triangular to triangular: Xo ∼ T (0, 1, 0.7), k = 3 δ = 0.5, k = 3

0.8

0.7

0.6

0.5

0.4

0.3 Probability density 0.2

0.1

0 −3 −2 −1 0 1 2 3 x (c) Gaussian to Gaussian: Xo ∼ N (0, 1), δ = 0.3, k = 1

Figure 3-2: Examples of reasonable uniform, triangular, and Gaussian distributions. In each ﬁgure, the solid red line denotes fXo (x), the dashed blue lines represent various possibilities for fX0 (x), and the dotted black lines show κfXo (x), which is greater than or equal to the corresponding density fX0 (x) for all x.

65 In the uniform to uniform and triangular to triangular cases, a key distinction between the current work and the procedures proposed by Allaire is that the k distributions for X0 are made to be evenly spaced on the interval [ao, bo]; this is to ensure the inclusion of a wide spectrum of possible distributions when estimating the adjusted main eﬀect sensitivity index.

Uniform to uniform: Xo ∼ U[ao, bo] → X0 ∼ U[a0, b0]

Selecting a0 and b0: b0−a0 2 0 0 1. From δ = bo−ao , solve for extent L = b − a . 2. Evenly space k samples of b0 on the interval [ao + L, bo].

3. For each sample of b0, ﬁnd a0 = b0 − L.

Specifying κ: bo−ao 1. Set κ = b0−a0 .

For triangular distributions, the variance is not solely a function of extent, but √ also of the peak location. For a factor Xo, the extent can vary between 18δσo2 and √ 24δσo2. The former case occurs when the peak coincides with either the lower or upper bound (i.e., co = ao or co = bo), resulting in a skewed distribution. The latter case corresponds to a symmetric triangular distribution in which the peak coincides with the mean value (i.e., co = (ao + bo)/2).

Triangular to triangular: Xo ∼ T (ao, bo, co) → X0 ∼ T (a0, b0, c0)

Selecting a0, b0, and c0: 0 1. Compute the minimum and maximum possible extent for X : Lmin = √ √ o2 o2 18δσ and Lmax = 24δσ .

2. Evenly space k samples of L on the interval [Lmin,Lmax].

3. For each sample of L, randomly sample midpoint M from the interval [ao, bo].

0 L 0 L • Find a = M − 2 and b = M + 2 .

66 • If a0 ≥ ao and b0 ≤ bo, accept a0 and b0. Otherwise, repeat Step 3.

a02+b02+c02−a0b0−a0c0−b0c0 0 4. From δ = ao2+bo2+co2−aobo−aoco−boco , solve for c . Select the root that is closer to co (this is to circumvent the problem of sample impoverishment towards the tails of the distribution).

Specifying κ: 0 0 1. Using (1.6), evaluate fX0 (c ) and fXo (c ).

0 fX0 (c ) 2. Set κ = 0 . fXo (c )

Gaussian to Gaussian: Xo ∼ N (µo, σo) → X0 ∼ N (µ0, σ0)

Selecting µ0 and σ0: 1. Set µ0 = µo. √ 2. Set σ0 = δσo2.

Specifying κ: 0 0 1. Using (1.7), evaluate fX0 (c ) and fXo (c ).

0 fX0 (µ ) 2. Set κ = 0 . fXo (µ )

3.2.2 Cases Involving Truncated Extent for Xo

In this section, we consider three cases in which it is not always feasible to generate samples of X0 from the entire distribution for Xo. These cases are triangular to uniform, Gaussian to uniform, and Gaussian to triangular, which form the lower left entries of Table 3.1. In the first case, the probability density of a triangular random variable Xo equals zero at ao and bo; thus, there are very few samples of Xo in the tails of the distribution, making it nearly impossible to generate samples of X0 from those regions using the AR method. Similarly, in the second and third cases, Xo is Gaussian and has infinite extent; however, sample impoverishment in the tails again limits the feasibility of the AR method to generate sufficiently many samples of X0.

67 To circumvent this problem, we instead use the AR method to generate samples of X0 from a truncated approximation for Xo. We specify lower and upper bounds 0 (BL and BU , respectively) for the interval on which X can be supported, such that o the vast majority of the probability mass of X is contained in [BL,BU ]. The lower and upper bounds can be computed for a desired level using the cumulative distribution function FXo (x), which is equal to the area under the probability density fXo (x) from −∞ to x:

o FXo (x) = P (X ≤ x) (3.11) Z x = fXo (ξ) dξ. (3.12) −∞

As an example, to specify the bounds [BL,BU ] that delimit the middle 95% of the probability mass of Xo, we can simply solve (3.12) for the values of x corresponding to FXo (x) = 0.025 and FXo (x) = 0.975, respectively. The analytical expressions for the cumulative distribution function of triangular and Gaussian random variables are given in (3.13) and (3.14), respectively:

 (x − a)2  if a ≤ x ≤ c, (b − a)(c − a)   2 o o o o  (b − x) X ∼ T (a , b , c ): FXo (x) = 1 − if c < x ≤ b, (3.13) (b − a)(b − c)    0 otherwise.

o o o o 1 x − µ X ∼ N (µ , σ ): FXo (x) = 1 + erf √ . (3.14) 2 2σo2

Empirical evidence suggests that reasonable bounds for the three cases in this

section (and the percentage of the area under fXo (x) that is captured), are:

−1 −1 • Triangular to uniform: BL = FXo (0.025), BU = FXo (0.975); middle 95%.

o o • Gaussian to uniform: [BL,BU ] = µ ∓ 2σ ; middle 95.4%.

o o • Gaussian to triangular: [BL,BU ] = µ ∓ 3σ ; middle 99.7%.

68 The procedures to estimate the parameters of X0 and specify κ for AR sampling are listed below. Examples of reasonable distributions for these three cases are shown in Figure 3-3 for select values of δ and k.

12 3

10 2.5

8 2

6 1.5

4 1 Probability density Probability density

2 0.5

0 0 0 0.2 0.4 0.6 0.8 1 −2 −1.5 −1 −0.5 0 0.5 1 1.5 2 x x (a) Triangular to uniform: Xo ∼ T (0, 1, 0.3), (b) Gaussian to uniform: Xo ∼ U[0, 1], δ = 0.7, δ = 0.5, k = 2 k = 2

1.4

1.2

0.8

0.6

Probability density 0.4

0.2

0 −3 −2 −1 0 1 2 3 x (c) Gaussian to triangular: Xo ∼ N (0, 1), δ = 0.6, k = 2

Figure 3-3: Examples of reasonable uniform, triangular, and Gaussian distributions for cases involving a truncated extent for Xo

69 Triangular to uniform: Xo ∼ T (ao, bo, co) → X0 ∼ U[a0, b0]

Selecting a0 and b0: −1 −1 1. Using (3.13), ﬁnd bounds BL = FXo (0.025) and BU = FXo (0.975).

3(b0−a0)2 0 0 2. From δ = 2(ao2+bo2+co2−aobo−aoco−boco) , solve for extent L = b − a .

0 3. Evenly space k samples of b on the interval [BL + L, BU ].

4. For each sample of b0, ﬁnd a0 = b0 − L.

Specifying κ: 0 0 1. Using (1.6), evaluate fXo (a ) and fXo (b ).

0 0 1 1 2. If fXo (a ) ≤ fXo (b ), set κ = 0 0 0 . Otherwise, set κ = 0 0 0 . fXo (a )(b −a ) fXo (b )(b −a )

Gaussian to uniform: Xo ∼ N (µo, σo) → X0 ∼ U[a0, b0]

Selecting a0 and b0: o o o o 1. Set lower bound BL = µ − 2σ and upper bound BU = µ + 2σ .

(b0−a0)2 0 0 2. From δ = 12σo2 , solve for extent L = b − a .

0 3. Evenly space k samples of b on the interval [BL + L, BU ].

4. For each sample of b0, ﬁnd a0 = b0 − L.

Specifying κ: 0 0 1. Using (1.7), evaluate fXo (a ) and fXo (b ).

0 0 1 1 2. If fXo (a ) ≤ fXo (b ), set κ = 0 0 0 . Otherwise, set κ = 0 0 0 . fXo (a )(b −a ) fXo (b )(b −a )

Gaussian to triangular: Xo ∼ N (µo, σo) → X0 ∼ T (a0, b0, c0)

Selecting a0, b0, and c0: 0 1. Compute the minimum and maximum possible extent for X : Lmin = √ √ o2 o2 18δσ and Lmax = 24δσ .

70 2. Evenly space k samples of L on the interval [Lmin,Lmax].

o o o o 3. Set lower bound BL = µ − 3σ and upper bound BU = µ + 3σ .

4. For each sample of L, randomly sample midpoint M from the interval

[BL,BU ].

0 L 0 L • Find a = M − 2 and b = M + 2 . 0 0 0 0 • If a ≥ BL and b ≤ BU , accept a and b . Otherwise, repeat Step 4.

a02+b02+c02−a0b0−a0c0−b0c0 0 5. From δ = 18σo2 , solve for c . Select the root that is closer to µo.

Specifying κ: 0 0 0 0 1. If a 6= c , ﬁnd αL and a < xL < c such that αLfXo (x) and fX0 (x) are 0 tangent at xL. Otherwise, set xL = a .

0 0 0 0 2. If b 6= c , ﬁnd αR and c < xR < b such that αRfXo (x) and fX0 (x) are 0 tangent at xR. Otherwise, set xR = b .

0 fX0 (xL) fX0 (xR) fX0 (c ) 3. Using (1.6) and (1.7), ﬁnd κL = , κR = , and κC = 0 . fXo (xL) fXo (xR) fXo (c )

4. Set κ = max{κL, κR, κC }.

3.2.3 Cases Involving Resampling

Finally, we consider the uniform to triangular, uniform to Gaussian, and triangular to Gaussian cases, which comprise the upper right corner of Table 3.1. In these cases, AR sampling may not be a valid method to generate samples of X0 from the existing samples of Xo. This is because for larger values of δ, although var (X0) < var (Xo), the extent of X0 can actually exceed that of Xo. For example, in Figure 3-4(a), Xo is a uniform random variable with variance σo2; two possibilities for X0 are shown, where X0 is a triangular random variable and δ = 0.9. In both cases the new extent b0 − a0 exceeds original extent bo − ao, making it impossible to generate samples in the tail regions of X0 through AR sampling. Therefore, in order to compute the adjusted main

71 effect sensitivity index for δ = 0.9, we must perform a new Monte Carlo simulation to generate samples of X0 and propagate them through the appropriate analysis models to obtain estimates of Y . √ o2 Recall that the minimum extent for a triangular distribution is Lmin = 18δσ ; for Lmin to be wholly encompassed within the bounds of a uniform distribution on [ao, bo] requires that δ ≤ 2/3. Thus, in the uniform to triangular case, AR sampling is feasible for δ ≤ 2/3; otherwise, resampling is necessary. For the uniform to Gaussian and triangular to Gaussian cases, the extent necessarily increases between Xo and X0 due the Gaussian distribution having infinite support. Here, we employ the same strategy as in Section 3.2.2 by using a truncated distribution to approximate X0 and ensuring that it lies entirely within the interval [ao, bo]. By requiring that ao ≤ µ0 − 3σ0 and bo ≥ µ0 + 3σ0, this approximation is sufficient to capture more than 99.7% of the probability mass of X0. This condition is satisifed for δ ≤ 1/3 in the uniform to Gaussian case; in the triangular to Gaussian case, the corresponding threshold varies between 1/2 and 2/3, depending on the skew- ness of the triangular distribution for Xo. If δ exceeds the specified value, resampling must be used in lieu of the AR method to generate samples of X0. For the three cases in this section, the procedures for specifying the parameters of X0 and generating samples from the distribution are provided below. Examples of reasonable distributions for these cases are shown in Figure 3-4 for select values of δ and k.

72 3 3

2.5 2.5

2 2

1.5 1.5

1 1 Probability density Probability density

0.5 0.5

0 0 −0.2 0 0.2 0.4 0.6 0.8 1 1.2 −0.2 0 0.2 0.4 0.6 0.8 1 1.2 x x (a) Uniform to triangular: Xo ∼ U[0, 1], δ = 0.9, (b) Uniform to triangular: Xo ∼ U[0, 1], δ = 0.4, k = 2 k = 3

3 3

2.5 2.5

2 2

1.5 1.5

1 1 Probability density Probability density

0.5 0.5

0 0 −0.2 0 0.2 0.4 0.6 0.8 1 1.2 −0.2 0 0.2 0.4 0.6 0.8 1 1.2 x x (c) Uniform to Gaussian: Xo ∼ U[0, 1], δ = 0.7, (d) Uniform to Gaussian: Xo ∼ U[0, 1], δ = 0.3, k = 1 k = 1

6 6

5 5

4 4

3 3

2 2 Probability density Probability density

1 1

0 0 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 x x (e) Triangular to Gaussian: Xo ∼ T (0, 1, 0.8), (f) Triangular to Gaussian: Xo ∼ T (0, 1, 0.8), δ = 0.5, k = 1 δ = 0.2, k = 1

Figure 3-4: Examples of reasonable uniform, triangular, and Gaussian distributions for cases involving resampling. The left column shows instances where δ is too large, and resampling is required. The right column shows instances where δ is suﬃciently small, and AR sampling can be used to generate samples of X0 from samples of Xo.

73 Uniform to triangular: Xo ∼ U[ao, bo] → X0 ∼ T (a0, b0, c0)

Selecting a0, b0, and c0: √ √ o2 o2 1. Compute Lmin = 18δσ and Lmax = 24δσ .

2. Sample L and set lower and upper bounds for sampling M.

• If δ ≤ 2/3:

o – Evenly space k samples of L on the interval [Lmin, min{Lmax, b − ao}]. o o – Set lower bound BL = a and upper bound BU = b . • If δ > 2/3:

– Evenly space k samples of L on the interval [Lmin,Lmax]. o o – Set lower bound BL = b −Lmax and upper bound BU = a +Lmax.

3. For each sample of L, randomly sample midpoint M from the interval

[BL,BU ].

BL+BU • If L = BU − BL, set M = 2 . 0 L 0 L • Find a = M − 2 and b = M + 2 . 0 0 0 0 • If a ≥ BL and b ≤ BU , accept a and b . Otherwise, repeat Step 3.

2(a02+b02+c02−a0b0−a0c0−b0c0) 0 4. From δ = 3(bo−ao)2 , solve for c . Select either root at random. Specifying κ: o o 0 • If δ ≤ 2/3, set κ = (b −a )fX0 (c ) and use AR sampling to generate samples of X0 ∼ T (a0, b0, c0) from samples of Xo ∼ U[ao, bo].

• If δ > 2/3, use MC sampling to generate samples of X0 ∼ T (a0, b0, c0).

74 Uniform to Gaussian: Xo ∼ U[ao, bo] → X0 ∼ N (µ0, σ0)

Selecting µ0 and σ0: 0 ao+bo 1. Set µ = 2 . q 0 δ(bo−ao)2 2. Set σ = 12 . Specifying κ: o o 0 • If δ ≤ 1/3, set κ = (b −a )fX0 (µ ) and use AR sampling to generate samples of X0 ∼ N (µ0, σ0) from samples of Xo ∼ U[ao, bo].

• If δ > 1/3, use MC sampling to generate samples of X0 ∼ N (µ0, σ0).

Triangular to Gaussian: Xo ∼ T [ao, bo, co] → X0 ∼ N (µ0, σ0)

Selecting µ0 and σ0: 0 ao+bo+co 1. Set µ = 3 . q 0 δ(ao2+bo2+co2−aobo−aoco−boco) 2. Set σ = 18 . Specifying κ: • If ao ≤ µ0 − 3σ0 and bo ≥ µ0 + 3σ0:

∗ 1. Using (1.6) and (1.7), ﬁnd x and α such that αfXo (x) and fX0 (x) are tangent at x∗.

∗ fX0 (x ) 0 2. Set κ = ∗ and use AR sampling to generate samples of X ∼ fXo (x ) N (µ0, σ0) from samples of Xo ∼ T [ao, bo, co].

• Otherwise, use MC sampling to generate samples of X0 ∼ N (µ0, σ0).

3.3 Local Sensitivity Analysis

In this section, we explore how complexity and risk estimates are aﬀected by small changes in the mean or standard deviation of a system’s design parameters or QOI. Revisiting Figures 1-4(a) and 1-4(b), we consider the separate cases of a constant

75 scaling of fY (y) that shrinks the standard deviation versus a constant shift in fY (y) that alters the mean value. These design activities have direct implications for system robustness and reliability. The results derived herein represent local sensitivities obtained by linearizing the system about the current estimate.

3.3.1 Local Sensitivity to QOI Mean and Standard Deviation

To evaluate the local sensitivity of complexity and risk with respect to design activities that perturb the mean (µY ) or standard deviation (σY ) of the QOI Y , we compute the partial derivative of C(Y ) and P (Y < r) with respect to µY and σY . The partial derivatives can then be linearized about the current design to predict the change in complexity (∆C) or risk (∆P ) associated with a perturbation in mean (∆µY ) or standard deviation (∆σY ). These expressions are derived in Appendix B; the key results are summarized in Table 3.2.

Sensitivity of C(Y ) with respect to µY and σY Partial derivatives ∂C(Y ) ∂C(Y ) C(Y ) = 0 (3.15) = (3.16) ∂µY ∂σY σY Local approximations ∆σ ∆C(Y ) = 0 (3.17) ∆C ≈ Y C(Y ) (3.18) σY

Sensitivity of P (Y < r) with respect to µY and σY Partial derivatives

∂P (Y < r) ∂P (Y < r) (µY − r) = −fY (r) (3.19) = fY (r) (3.20) ∂µY ∂σY σY Local approximations

∆σY ∆P ≈ −∆µY fY (r) (3.21) ∆P ≈ (µY − r)fY (r) (3.22) σY

Table 3.2: Local sensitivity analysis results for complexity and risk

From (3.15), we see that perturbations in µY do not aﬀect complexity. This is

76 unsurprising, as we know that C(Y ) contains no information about the expected value of Y . From (3.16), we see that the sensitivity of complexity to σY is a constant, and simply equals the ratio of C(Y ) to σY . Since both C(Y ) and σY are non-negative, this implies that as a local approximation (see (3.18)), reducing standard deviation also reduces complexity.

More interestingly, (3.19) and (3.20) show that the sensitivity of risk to both µY and σY is proportional to the probability density of Y evaluated at the requirement r. Since fY (y) ≥ 0 for all values of y, this implies that the sign of ∆µY or µY − r determines whether risk is increased or decreased (see (3.21) and (3.22)). Figure 3-5 helps to illustrate this point for (3.22), where a reduction in standard deviation can alter risk in either direction, depending on the relative locations of r and µY . Refinement: Change Std Dev Refinement: Change Std Dev

푓푌 푦 푓푌 푦 0.8 0.8

0.6 0.6 (x)

(x) 0.4 0.4

f f

0.2 푃 푌 < 푟 0.2 푃 푌 < 푟

0 푦0 푦 푟 휇푌 휇푌 푟

-0.2 (a) µY − r > 0 -0.2 (b) µY − r < 0 -4 -3 -2 -1 0 1 2 3 -4 4 -3 -2 -1 0 1 2 3 4 x x Figure 3-5: The relative locations of r and µY greatly impact the change in risk associated with a decrease in σY . Moving from the red probability density to the blue, P (Y < r) decreases if µY − r > 0, and increases if µY − r < 0.

3.3.2 Relationship to Variance-Based Sensitivity Indices

In this section, we seek to relate the local approximations in (3.18) and (3.22) to sensitivity indices computed from variance-based GSA and DSA. We let Y o and Y 0 represent initial and new estimates of the QOI (the red and blue densities in Figure 3-5, respectively) corresponding to a design activity that reduces variance (and thus standard deviation). Similarly, we let var (Y o) and var (Y 0) represent the variance of

77 the QOI before and after the update, respectively. If the activity is one that results in learning the true value of factor i, then we know from (3.7) that:

0 o o var (Y ) = var (Y ) − Sivar (Y ). (3.23)

The ratio of var (Y 0) to var (Y o) is given by:

var (Y 0) = 1 − S . (3.24) var (Y o) i

We can relate the above ratio to the quantity ∆σY /σY from (3.18) and (3.22):

p 0 p o ∆σY var (Y ) − var (Y ) = p σY var (Y o) s var (Y 0) = − 1 var (Y o) p = 1 − Si − 1. (3.25)

This allows us to rewrite (3.18) and (3.22) as:

p ∆C ≈ ( 1 − Si − 1)C(Y ), (3.26) p ∆P ≈ ( 1 − Si − 1)(µY − r)fY (r), (3.27)

noting that C(Y ), µY , and fY (r) in the above expressions refer to the complexity, mean, and probability density (evaluated at r) of the QOI for the initial design. Finally, if the design activity is instead one that reduces the variance of factor i by 100(1 − δ)%, then (3.23) can be modiﬁed to:

0 o o var (Y ) = var (Y ) − adjSi(δ)var (Y ), (3.28)

and we can similarly substitute adjSi(δ) for Si in (3.26) and (3.27).

Since both Si and adjSi(δ) can only assume values in the interval [0, 1], the above local approximations imply that a decrease in variance is concurrent with a reduction

78 in complexity, as well as in risk if µY − r > 0. While an attractive result, these relations are simply local approximations and should not be generalized. As we will demonstrate in Chapter 4, variance, complexity, and risk each describes a diﬀerent aspect of system uncertainty, and can therefore trend in opposite directions.

3.3.3 Local Sensitivity to Design Parameter Mean and Stan- dard Deviation

In order to use sensitivity information about the QOI to update the design, it is necessary to translate this information into tangible actions for modifying the design parameters. For this, we extend (3.19) and (3.20) to compute the sensitivity of risk to perturbations in the mean or standard deviation of the design parameters.

T T We deﬁne the vectors µX = [µX1 , µX2 , . . . , µXm ] and σX = [σX1 , σX2 , . . . , σXm ] , which consist of the mean and standard deviation estimates of the entries of X. Using the chain rule, the partial derivative of P (Y < r) with respect to µXi is given by:

∂P (Y < r) ∂P (Y < r) ∂µ = Y . (3.29) ∂µXi ∂µY ∂µXi

To estimate ∂µY /∂µXi , we use (1.4) and make the following approximation for µY :

µY = E [g(X)] ≈ g(E [X]) = g(µX). (3.30)

This approximation is exact if g(X) is linear. Therefore, we have:

∂µ ∂g(µ ) Y ≈ X . (3.31) ∂µXi ∂Xi

Combining (3.19), (3.29), and (3.31), we obtain:

∂P (Y < r) ∂g(µX) ≈ −fY (r) , (3.32) ∂µXi ∂Xi ∂g(µX) ∆P ≈ −∆µXi fY (r) . (3.33) ∂Xi

79 Similarly, the partial derivative of P (Y < r) with respect to σXi is given by:

∂P (Y < r) ∂P (Y < r) ∂σ = Y . (3.34) ∂σXi ∂σY ∂σXi

To approximate ∂σY /∂σXi , we can use results from DSA to estimate ∆σY /∆σXi . The

adjusted main eﬀect sensitivity index adjSi(δ) relates the expected variance remaining in Y (given by (3.28)) to the variance remaining in factor i (given by (3.10)) as δ

ranges between 0 and 1, which allows us to compute ∆σY /∆σXi as follows:

∆σ pvar (Y o) − adjS (δ)var (Y o) − pvar (Y o) Y = i p o p o ∆σXi δvar (Xi ) − var (Xi ) σ (p1 − adjS (δ) − 1) = Y √ i . (3.35) σXi ( δ − 1)

Thus, we can combine (3.20), (3.34) and (3.35) to obtain:3

∂P (Y < r) f (r)(µ − r)(p1 − adjS (δ) − 1) ≈ Y Y √ i , (3.36) ∂σ Xi σXi ( δ − 1) ∆σ f (r)(µ − r)(p1 − adjS (δ) − 1) ∆P ≈ Xi Y Y √ i . (3.37) σXi ( δ − 1)

The key result of this section is that we can relate system risk to the speciﬁc changes in the design parameters. As an example, let ∆P denote the desired change in risk. Rearranging (3.33) and (3.37), we can estimate the requisite change in the mean and standard deviation of each design parameter in order to achieve that goal. For small values of ∆P , these changes are approximated by:

∆P ∂g(µ )−1 ∆µ ≈ − X , (3.38) Xi f (r) ∂X Y i √ ∆P σ ( δ − 1) ∆σ ≈ Xi . (3.39) Xi p fY (r)(µY − r)( 1 − adjSi(δ) − 1)

3 While (3.35) is a valid statement of ∆σY /∆σXi for any value of δ ∈ [0, 1), the choice of δ can

greatly aﬀect the accuracy of the local approximation ∂σY /∂σXi ≈ ∆σY /∆σXi , especially for small values of δ (corresponding to large reductions in factor variance), or if the relationship between 0 0 var (Y ) and var (Xi) is highly nonlinear.

80 The above expressions allow the designer to obtain a first-order estimate of the parameter adjustments needed to achieve a desired decrease in risk. They can also highlight different trends, tradeoffs, and design tensions present in the system. However, we note again that these relations are merely local approximations whose predictive accuracy cannot be guaranteed. Furthermore, they do not account for interactions among the design parameters, nor do they imply that the suggested changes in mean or standard deviation are necessarily feasible, as limitations due to physical or budgetary constraints are not accounted for. It is the responsibility of the designer to use these tools as a guideline for cost-benefit analysis of various design activities, and ultimately select the most appropriate action for risk mitigation.

3.4 Entropy Power-Based Sensitivity Analysis

In this section, we extend the ideas of variance-based GSA to an analogous methodology for complexity-based GSA. For this, we use entropy power as the basis for our sensitivity analysis approach, recalling from (2.27) that entropy power is proportional to the square of exponential entropy.

3.4.1 Entropy Power Decomposition

We rewrite the ANOVA-HDMR presented in (3.1) using the random variables Y and

Zi,Zij,...,Z12...m to denote the outputs of the original function and various sub- functions, respectively. That is, we let Y = g(X) and Zi1,...,is = gi1,...,is (Xi1 ,...,Xis ), where the indices i1, . . . , is represent all sets of s integers that satisfy 1 ≤ i1 < . . . < is ≤ m, as described in Section 3.1.1. The ANOVA-HDMR can thus be expressed as:

X X Y = g0 + Zi + Zij + ... + Z12...m. (3.40) i i

Since all sub-functions in (3.1) are orthogonal and zero-mean, Zi,Zij,...,Z12...m (henceforth termed “auxiliary random variables”) are all uncorrelated. According to (3.5), the variance of Y is the sum of the partial variances of Z1 through Zm and

81 their higher-order interactions:

X X var (Y ) = var (Zi) + var (Zij) + ... + var (Z12...m). (3.41) i i

Normalizing by var (Y ), the proportional contribution to var (Y ) from each of the auxiliary random variables is given by:

X var (Zi) X var (Zij) var (Z12...m) 1 = + + ... + . (3.42) var (Y ) var (Y ) var (Y ) i i

We desire to derive a similar relationship for N(Y ), the entropy power of the QOI. As discussed in Section 2.3.3, although there is no general relationship between entropy power and variance, for a given random variable, entropy power is always less than or equal to variance, with equality in the case where the random variable is Gaussian (see Table 2.1). Therefore, we seek to relate the disparity between variance and entropy power to the random variable’s degree of non-Gaussianity, quantified using the K-L divergence. For this, we define an “equivalent Gaussian distribution,” which is a Gaussian random variable with the same mean and variance as the original random variable. For an arbitrary random variable W , we denote its equivalent G Gaussian distribution using the superscript G, and define it as W ∼ N (µW , σW ).

Using (2.22), we compute the K-L divergence between each random variable in G G (3.40) and its equivalent Gaussian distribution to obtain DKL(Y ||Y ), DKL(Z1||Z1 ), G DKL(Z2||Z2 ), etc. In Appendix C, we derive the following sum, which relates the entropy power and non-Gaussianity of the QOI to the corresponding quantities for the auxiliary random variables:

G X G N(Y )exp[2DKL(Y ||Y )] = N(Zi) exp[2DKL(Zi||Zi )] i X G + N(Zij) exp[2DKL(Zij||Zij )] + ... i

G + N(Z12...m) exp[2DKL(Z12...m||Z12...m)]. (3.43)

82 The basis for (3.43) is that for any arbitrary random variable W , var (W ) = G 4 N(W ) exp[2DKL(W ||W )]. This implies that the variance of each of Y and Zi,Zij,

...,Z12...m is equal to the product of the variable’s entropy power and the exponential term, where the argument of the exponential is two times the K-L divergence of the variable with respect to its equivalent Gaussian distribution. Thus, (3.41) and (3.43) are equivalent.

Note that in the derivation we have not made any assumptions about the underlying distributions of Y or the auxiliary random variables. The only requirement is that all the auxiliary random variables must be uncorrelated, which is automatically satisfied by the ANOVA-HDMR. We emphasize the distinction between uncorrelated random variables Zi and Zj (i 6= j), for which the variance of the sum is equal to the sum of the variances (i.e., var (Zi + Zj) = var (Zi) + var (Zj)), and independent random variables Xi and Xj, for which the joint probability density fXi,Xj (xi, xj) is equal to the product of the marginal densities fXi (xi) and fXj (xj). While independent implies uncorrelated, the converse is not true. Recall that in Section 1.2 we made the assumption that all design parameters X1,...,Xm are independent. On the other hand, the auxiliary random variables Z1,Z2,...,Z12...m are uncorrelated. While the auxiliary random variables representing main effects (e.g., Z1,Z2,...,Zm) are also mutually independent, those that correspond to interaction effects are functions of two or more design parameters, and thus are necessarily dependent. For example, while Zi and Zj, i 6= j, are independent, Zij is dependent on both Zi and Zj, but all three variables are uncorrelated.

Having obtained analogous expressions for the decomposition of output variance and entropy power into comprising terms, we can normalize (3.43) to establish a similar interpretation in terms of proportional contributions from the input factors

4This result is consistent with the principle of maximum entropy, which states that for a given mean and variance, diﬀerential entropy (and therefore entropy power) is maximized with a Gaussian G distribution. Since the K-L divergence is a non-negative quantity, exp[2DKL(W ||W )] ≥ 1, which implies that var (W ) ≥ N(W ). In the limiting case that W is itself a Gaussian random variable, G G DKL(W ||W ) = 0, exp[2DKL(W ||W )] = 1, and var (W ) = N(W ), as required.

83 and their interactions:

G G X N(Zi) exp[2DKL(Zi||Z )] X N(Zij) exp[2DKL(Zij||Zij )] 1 = i + N(Y ) exp[2D (Y ||Y G)] N(Y ) exp[2D (Y ||Y G)] i KL i

In (3.44), the proportion of output uncertainty directly due to Xi consists of two parts: one that is the ratio of the entropy power of Zi to that of Y , and one that is the G ratio of the exponential of twice the K-L divergence from Zi to Zi to the analogous quantity for Y . Recalling that exponential entropy — proportional to the square root of entropy power — measures the intrinsic extent of a random variable, we conclude that the ﬁrst ratio is directly inﬂuenced by the intrinsic extent of Xi (and thus Zi).

On the other hand, the second ratio is directly related to the non-Gaussianity of Xi

(and thus Zi). To illustrate this, we introduce the entropy power sensitivity indices

ηi and ζi, deﬁned as:

N(Z ) η = i , (3.45) i N(Y ) G exp[2DKL(Zi||Zi )] ζi = G . (3.46) exp[2DKL(Y ||Y )]

The above expressions correspond to the main eﬀect indices for factor i; analogous expressions can also be derived for the higher-order interactions. Due to the equiv- alence of (3.42) and (3.44), ηi and ζi are related to the variance-based main eﬀect sensitivity index Si as follows:

Si = ηiζi. (3.47)

Substituting ηi and ζi into (3.44) gives:

1 =η1ζ1 + η2ζ2 + ... + η12...mζ12...m. (3.48)

The proportion of output uncertainty due to each factor directly can be broken down

84 into an intrinsic extent effect characterized by the sensitivity index ηi, which is related to complexity, and a non-Gaussianity effect characterized by the sensitivity index ζi, which is related to distribution shape. The product of the two effects equals the MSI of the factor. This allows us to associate (3.44) with the uncertainty apportionment notion depicted in Figure 3-1.

Note that in the previous paragraph we used the phrase “due to each factor directly” instead of “due to each factor alone.” This choice of wording reflects the fact that a change in any auxiliary random variable Zj can affect both the intrinsic G extent and the non-Gaussianity of Y . Thus, the quantities N(Y ) and DKL(Y ||Y ) are impacted, which indirectly affects the indices ηi and ζi of all other auxiliary variables

Zi where i 6= j. Because it is usually diﬃcult to determine a priori how changing G Zj would modify N(Y ) and DKL(Y ||Y ), it is typically impractical to decouple the direct and indirect eﬀects that alterations to Zj would impose on the entropy power decomposition. Despite this limitation, the entropy power sensitivity indices still reveal useful information about how the spread and distribution shape of each input factor contribute to uncertainty in the output quantity of interest.

3.4.2 The Eﬀect of Distribution Shape

The K-L divergence between Zi and its equivalent Gaussian distribution is both shift- and scaling-invariant. The former is easy to comprehend, since by deﬁnition Zi and G 0 Zi have the same mean. To see the latter, we examine the case where Zi = αZi. It 0 is easy to show that h(Zi) = h(Zi) + log α (see Table 2.2) [88]. Making use of the 1 z z 0 0G relations f 0 (z) = f ( ) and ξ = , we can compute the cross entropy h(Z ,Z ) Zi α Zi α α i i

85 as follows:

Z ∞ 0 0G 1 z 1 z h(Z ,Z ) = − fZ ( ) log f G ( ) dz i i i Zi −∞ α α α α Z ∞ 1 1 = − fZ (ξ) log f G (ξ) α dξ i Zi −∞ α α Z ∞ Z ∞ = − fZ (ξ) log f G (ξ) dξ + log α fZ (ξ) dξ i Zi i −∞ −∞ G = h(Zi,Zi ) + log α. (3.49)

Combining (2.24) and (3.49), we obtain:

0 0G G DKL(Zi||Zi ) = DKL(Zi||Zi ). (3.50)

The above result implies that the non-Gaussianity of a random variable remains constant if its underlying distribution maintains the same shape to within a multi- 0 plicative constant. For example, if Zi and Zi are both uniform random variables, but 0 Zi has a narrower distribution, the fundamental shape of probability density is not affected, and therefore the two variables have the same degree of non-Gaussianity. 0 However, if Zi and Zi are both triangular random variables, but one has a symmetric 0 distribution and the other a skewed distribution, then Zi and Zi have differing levels of non-Gaussianity. This is because the underlying distribution shape has changed, even though both variables are of the triangular family. Finally, as a note of caution, G we reiterate that even if a design activity has no effect on DKL(Zi||Zi ), ζi can still G be impacted indirectly through changes to DKL(Y ||Y ) imparted by other factors.

Since it is a measure of distance, the K-L divergence is always non-negative. As a G measure of non-Gaussianity, DKL(Zi||Zi ) = 0 if and only if Zi itself is Gaussian, and G is positive otherwise. This implies that exp[2DKL(Zi||Zi )] ≥ 1. However, ζi can be greater than, less than, or equal to one, depending on the relative magnitude of the numerator and denominator in (3.46). That is to say, ζi indicates whether an auxiliary random variable Zi is less Gaussian (ζi > 1), more Gaussian (ζi < 1), or equally as

Gaussian (ζi = 1) as the QOI Y . Below, we consider the three cases separately for a

86 system modeled by Y = X1 + X2, where X1 and X2 are either uniform or Gaussian random variables. Note that this simple system does not contain interaction eﬀects, and thus Z1 = X1, Z2 = X2, and Z12 = 0.

푿ퟏ~푼 −ퟎ. ퟓ, ퟎ. ퟓ 푿ퟐ~푼 −ퟎ. ퟓ, ퟎ. ퟓ 풀~푻 −ퟏ, ퟏ, ퟎ

-0.5 0 0.5 -0.5 0 0.5 -1 -0.5 0 0.5 1

(a) Case 1: Example with ζ1 > 1 and ζ2 > 1

2.5 2.5

푿ퟏ~푼 −ퟎ. ퟓ, ퟎ. ퟓ 2 푿ퟐ~푵 ퟎ, ퟎ. ퟐퟓ 2 풀

1.5 1.5

1 1

0.5 0.5

0 0 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 -1.5 -1 -0.5 0 0.5 1 1.5 -0.5 0 0.5 -0.5 0 0.5 -1 -0.5 0 0.5 1

(b) Case 2: Example with ζ1 > 1 and ζ2 < 1

2.5 2.5 푿 ~푵 ퟎ, ퟎ. ퟐퟓ 푿 ~푵 ퟎ, ퟎ. ퟐퟓ 풀~푵 ퟎ, ퟎ. ퟑퟓ 2 ퟏ ퟐ 2

1.5 1.5

1 1

0.5 0.5

0 0 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 -1.5 -1 -0.5 0 0.5 1 1.5 -0.5 0 0.5 -0.5 0 0.5 -1 -0.5 0 0.5 1

Figure 3-6: Examples of Y = X1 + X2 with increase, decrease, and no change in Gaussianity between the design parameters and QOI

For the ﬁrst case, we let X1 and X2 be i.i.d. uniform random variables on the interval [−0.5, 0.5]. The sum Y = X1 + X2 has a symmetric triangular distribution on the interval [−1, 1]. In this case, both ζ1 and ζ2 exceed one (Figure 3-7(a)). Figure 3-6(a) illustrates that in moving from the design parameters to the QOI, the system increases in Gaussianity. Figure 3-6(b) shows an example where X1 is uniform and

X2 is Gaussian, resulting in a distribution for Y that is more Gaussian than X1 but less Gaussian than X2. In this case, the corresponding indices for non-Gaussianity are ζ1 > 1 and ζ2 < 1 (Figure 3-7(b)). Finally, if both X1 and X2 are Gaussian, then

X1 + X2 is also Gaussian (Figure 3-6(c)), and ζ1 = ζ2 = 1 (Figure 3-7(c)).

87 1.6 1.6 S S 1.42 i 1.37 1.37 i 1.4 η 1.4 η i i 1.2 ζ 1.2 ζ i i 0.99 1 1

0.8 0.8 0.57 0.6 0.50 0.50 0.6 0.43 0.43 0.37 0.37 0.40 0.4 0.4

0.2 0.2

0 0 Factor 1 Factor 2 Factor 1 Factor 2

(a) Case 1: X1,X2 ∼ U[−0.5, 0.5] (b) Case 2: X1 ∼ U[−0.5, 0.5], X2 ∼ N (0, 0.25)

1.6 S i 1.4 η i 1.2 ζ i 1.00 1.00 1

0.8

0.6 0.50 0.50 0.50 0.50

0.4

0.2

0 Factor 1 Factor 2

Figure 3-7: Sensitivity indices Si, ηi, and ζi for three examples of Y = X1 + X2. For each factor, Si equals the product of ηi and ζi.

3.5 Chapter Summary

In the design of complex systems, sensitivity analysis is crucially important for understanding how various sources of uncertainty inﬂuence estimates of the quantities of interest. In this chapter, we built upon the existing methodology for variance-based distributional sensitivity analysis to consider changes in distribution family and thus explicitly study the eﬀect of distribution shape. We also computed the local sensitivity of complexity and risk to perturbations in mean and standard deviation, and interpreted the results in the context of design for robustness and reliability. Fur- thermore, we connected these sensitivities to indices from GSA and DSA to generate local approximations for the change in complexity or risk associated with a reduction in variance.

88 Another contribution of this chapter is the derivation of the entropy power decomposition, which categorizes uncertainty contributions into intrinsic extent and non-Gaussianity effects. The entropy power decomposition is analogous to the notion of variance apportionment in GSA; however, it provides additional information about how different aspects of a factor’s uncertainty are compounded in the system, which can be valuable to the designer when selecting specific strategies to reduce uncertainty. In Chapter 4, we will connect the sensitivity analysis methods from this chapter with design budgets to trade off various options for uncertainty mitigation and guide decisions for the allocation of resources.

Chapter 4

Application to Engineering System Design

In this chapter, we apply the framework shown in Figure 1-2 to a simple engineering system in order to showcase its utility for designing for robustness and reliability. Throughout this discussion, we will use as an example a R-C high-pass filter circuit, which is introduced in the Section 4.1. Section 4.2 demonstrates how the sensitivity analysis techiques developed in this thesis can be used to better our understanding of the effects of uncertainty in the system, as well as analyze the influence of various modeling assumptions. In Section 4.3, we introduce design budgets for cost and uncertainty, and connect them with sensitivity analysis results in order to identify specific options for uncertainty mitigation, visualize tradeoffs between various alternatives, and inform resource allocation decisions.

4.1 R-C Circuit Example

As an example system, consider the circuit shown in Figure 4-1, which consists of a resistor and a capacitor in series. The design parameters of the system are resistance (R) and capacitance (C), which have nominal values 100 Ω and 4.7 µF, respectively.

The corresponding component tolerances are Rtol = ±10% and Ctol = ±20%.

91 퐶

푉in 푅 푉out

Figure 4-1: R-C high-pass ﬁlter circuit

As configured, the circuit acts as a passive high-pass filter, whose magnitude and phase Bode diagrams are shown in Figure 4-2. We select the quantity of interest to be the cutoff frequency fc, which has a nominal value of 339 Hz. To achieve the desired

circuit performance, fc must exceed 300 Hz.

-10

-20

-30

Magnitude(dB) Magnitude(dB) -40

) deg

Phase(deg) Phase( 0 1 2 3 4 10 10 10 10 FrequencyFrequency (Hz)(Hz)

Figure 4-2: Bode diagrams for magnitude (top) and phase (bottom). The dashed black line indicates the required cutoﬀ frequency of 300 Hz.

We use the random variables X1, X2, and Y to represent uncertainty in R, C, and

fc, respectively. Due to stochasticity in the system, the functional requirement on the QOI will be treated as a probabilistic design target (i.e., the probability that the cutoﬀ frequency falls below 300 Hz must not exceed a speciﬁed limit). The relationship between the design parameters and the QOI is given by:

1 Y = . (4.1) 2πX1X2

92 We model X1 and X2 as uniform random variables on the intervals speciﬁed by their component tolerances, such that X1 ∼ U[a1, b1] = U[90, 110] Ω and X2 ∼

U[a2, b2] = U[3.76, 5.64] µF. For simplicity, we shall refer to this initial scenario as Case 0. For this system, we can evaluate the ANOVA-HDMR of (4.1) in closed form to obtain the following expressions for the expected value of Y , as well as the auxiliary random variables Z1, Z2, and Z12:

log(b1/a1) log(b2/a2) E [Y ] = , (4.2) 2π(b1 − a1)(b2 − a2)

log(b2/a2) Z1 = − E [Y ], (4.3) 2π(b2 − a2)X1

log(b1/a1) Z2 = − E [Y ], (4.4) 2π(b1 − a1)X2

1 log(b2/a2) Z12 = − 2πX1X2 2π(b2 − a2)X1 log(b /a ) log(b /a ) log(b /a ) − 1 1 + 1 1 2 2 . (4.5) 2π(b1 − a1)X2 2π(b1 − a1)(b2 − a2)

1400 2 1200 var(Y) = 2015 Hz C(Y) = 181 Hz 1000 P(Y<300Hz) = 18.6% 800

600

400 Number of samples

200

0 250 300 350 400 450 500 Cutoff frequency (Hz)

Figure 4-3: Histogram of cutoﬀ frequency for Case 0 generated using 10,000 MC samples. The dashed black line indicates the required cutoﬀ frequency of 300 Hz.

In this example, we use Monte Carlo simulation with 10,000 samples to characterize the propagation of uncertainty from the design parameters to the quantity of interest. Figure 4-3 shows the histogram of Y for Case 0. The corresponding uncertainty

93 estimates are: var (Y ) = 2015 Hz2, C(Y ) = 181 Hz, and P (Y < 300 Hz) = 18.6%.

4.2 Sensitivity Analysis of the R-C Circuit

4.2.1 Identifying the Drivers of Uncertainty

Having obtained uncertainty estimates for Case 0, we next apply existing variance- based sensitivity analysis methods to identify the key drivers of output variance. Global sensitivity analysis of Case 0 reveals that approximately 80% of the variability in Y can be attributed to X2 (capacitance), and the remaining 20% to X1 (resistance), with interactions playing a negligible role (Figure 4-4(a)). Distributional sensitivity analysis (assuming ﬁxed distribution shape) indicates that the relationship between a decrease in factor variance and the expected decrease in output variance is linear for both X1 and X2 (Figure 4-4(b)). These results point to prioritizing X2 for uncertainty reduction, as it dominates X1 in terms of contribution to output variance.

< 1% 1 Factor 1 0.8 Factor 2

20% 0.6 ) δ ( i 0.4 adjS 80% 0.2 Factor 1 Factor 2 0 Interactions

−0.2 0 0.2 0.4 0.6 0.8 1 1−δ (a) Main eﬀect sensitivity indices (b) Adjusted main eﬀect sensitivity indices

Figure 4-4: Global and distributional sensitivity analysis results for Case 0

4.2.2 Understanding the Impacts of Modeling Assumptions

In this section, we extend our sensitivity analysis of the R-C circuit to study the eﬀects of modeling assumptions, in particular assumptions regarding the input distributions.

94 Focusing on X2, we apply several methods developed in this thesis, including entropy power-based sensitivity analysis and DSA that permits changes to distribution family.

Two speciﬁc cases are considered: in Case 1, we allow the distribution of X2 to vary between several types while holding its variance constant; in Case 2, the interval on

which X2 is supported is ﬁxed while the distribution type is varied across several families. In both cases, we are interested in studying how assumptions regarding the input factor distribution impact uncertainty in the QOI.

Case 1: Fixing the Variance of X2

As described in Case 0, we initially assumed that X1 and X2 are uniformly distributed on the intervals [90, 110] Ω and [3.76, 5.64] µF, respectively, which correspond to to 2 2 variances of var (X1) = 33.3 Ω and var (X2) = 0.30 µF . Here, we use the procedures developed in Section 3.2 to extend DSA from Figure 4-4(b) to permit changes between distribution families.

1 1 Uni −> Uni Uni −> Uni Uni −> Tri Uni −> Tri 0.8 Uni −> Gaussian 0.8 Uni −> Gaussian Uni −> Bmd Uni Uni −> Bmd Uni 0.6 0.6 ) ) δ δ ( ( i 0.4 i 0.4 adjS adjS 0.2 0.2

0 0

−0.2 −0.2 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 1−δ 1−δ (a) Factor 1: Resistance (b) Factor 2: Capacitance

Figure 4-5: Distributional sensitivity analysis results with changes to distribution family

Figure 4-5(a) shows that for X1, there is virtually no diﬀerence in the adjusted main eﬀect sensitivity indices between the four scenarios considered: the initial case

where var (X1) is reduced while maintaining X1 as a uniform random variable, and

the alternative options where a decrease in var (X1) is concomitant with a switch to a triangular, Gaussian, or bimodal uniform distribution. This result suggests that

95 the shape of the underlying distribution for X1 does not contribute signiﬁcantly to

uncertainty in the QOI; that is, the relationship between a decrease in var (X1) and the expected reduction in var (Y ) is not aﬀected by the distribution family (of the four considered) to which X1 belongs.

Figure 4-5(b) shows a similar trend for X2; however, the four curves diverge slightly for small values of 1−δ. To study this region, we concentrate on the leftmost points in

Figure 4-5(b) where 1−δ = 0, which correspond to no reduction in var (X2) from Case

0. Figure 4-6 shows four possible distributions for X2, each of which has a variance of 0.30 µF2. Note that the triangular and Gaussian distributions have supports that extend beyond the original interval.1

1.5 U T G BU 1

0.5 Probability density

0 3 3.5 4 4.5 5 5.5 6 6.5 Capacitance (µF)

Figure 4-6: Examples of uniform (U), triangular (T), Gaussian (G), and bimodal 2 uniform (BU) distributions for X2, each with a variance of 0.30 µF

Case var (X2) C(X2) var (Y ) C(Y ) P (Y < 300 Hz) 1-T 0% +16.6% +2.1% +1.2% −1.5% 1-G 0% +19.3% +1.3% −1.9% −2.3% 1-BU 0% −50.0% −2.9% −4.5% +2.6%

Table 4.1: Change in uncertainty estimates for Cases 1-T, 1-G, and 1-BU, as compared to Case 0

1Although a Gaussian random variable is inﬁnitely supported, in this example we have truncated the Gaussian distribution at ±3 standard deviations (which captures more than 99.7% of the probability), as it is not realistic for capacitance to approach ±∞.

96 Table 4.1 shows how uncertainty estimates for the system are altered as a result

of changing the distribution of X2 from uniform (Case 0) to triangular (Case 1-T), truncated Gaussian (Case 1-G), or bimodal uniform (Case 1-BU). It is easy to see that estimates of variance, complexity, and risk can trend in opposite directions. For example, in both Cases 1-T and 1-G there is a signiﬁcant increase in C(X2), which results in a slight increase in output variance but a slight decrease in risk. Furthermore,

Case 1-G, which represents the maximum entropy distribution for X2 given a variance of 0.30 µF2, does not necessarily correspond to the most conservative estimates for output uncertainty. In Case 1-BU, the complexity of X2 is reduced by 50%, which in turn decreases both output variance and complexity but increases risk. These results demonstrate the presence of competing objectives for uncertainty mitigation, such that eﬀorts to reduce one measure of uncertainty can result in an increase in the others. In general, the extent to which the various measures of uncertainty trend jointly or divergently is problem-dependent.

Case 2: Fixing the Extent of X2

In Case 2, we investigate the effect of our initial assumption of a uniform distribution for X2 to capture the uncertainty associated with a ±20% capacitor tolerance. Figure 4-7 shows several alternative distributions on the interval [3.76, 5.64] µF: triangular and truncated Gaussian distributions centered at 4.7 µF (Cases 2-T and 2- G, respectively),2 and a symmetric bimodal uniform distribution parameterized by aL = 3.76 µF, bL = 4.23 µF, aR = 5.17 µF, and bR = 5.64 µF (Case 2-BU). Since these distributions lie wholly within the original interval, we can sample from them using the acceptance/rejection procedures provided in Section 3.2; thus, no additional MC simulations are required for the following analysis. The distributions shown in Figure 4-7 differ greatly in variance as well as complexity (see Table 4.2). Cases 2-T and 2-G can represent reasonable choices for X2 if there is strong confidence in the nominal capacitance. A possible scenario for Case 2-BU is

2As before, we approximate the Gaussian distribution with a truncated version in which ±3 standard deviations of the distribution fall within the speciﬁed interval.

97 1.5 U T G BU 1

0.5 Probability density

0 3 3.5 4 4.5 5 5.5 6 6.5 Capacitance (µF)

Figure 4-7: Examples of uniform (U), triangular (T), Gaussian (G), and bimodal uniform (BU) distributions for X2 supported on the interval [3.76, 5.64] µF

Case var (X2) C(X2) var (Y ) C(Y ) P (Y < 300 Hz) 2-T −50.0% −17.6% −41.2% −21.6% −7.4% 2-G −66.7% −31.1% −54.6% −30.8% −10.5% 2-BU +75.0% −50.0% +60.7% +11.9% +11.2%

Table 4.2: Change in uncertainty estimates for Cases 2-T, 2-G, and 2-BU, as compared to Case 0

one in which the capacitors with ±10% tolerance are removed during the manufacturing process and sold for a higher price, leaving the worst-performing components in the ±20% tolerance category. Figure 4-8 shows the histograms of Y generated using 10,000 MC samples for the 3 various cases. It is easy to see that the choice of X2 greatly aﬀects the shape of the resulting distribution for Y , as well as the corresponding uncertainty estimates. Table

4.2 shows that in Cases 2-T and 2-G, there is a signiﬁcant decrease in both var (X2) and C(X2) over Case 0, which also results in large reductions in the variance, complexity, and risk in the QOI. These outcomes are consistent with the adjusted main eﬀect sensitivity indices shown in Figure 4-5(b), which predict the expected reduction in var (Y ) to be approximately 40% and 55% for Cases 2-T and 2-G, respectively, corresponding to reductions in var (X2) of 50% and 66.7%. Case 2-BU, on the other hand,

3Note that Figure 4-3 is reproduced as Figure 4-8(a) as a point of comparison in Case 2.

98 1400 1400 2 2 1200 var(Y) = 2015 Hz 1200 var(Y) = 1182 Hz C(Y) = 181 Hz C(Y) = 142 Hz 1000 1000 P(Y<300Hz) = 18.6% P(Y<300Hz) = 10.9% 800 800

600 600

400 400 Number of samples Number of samples

200 200

0 0 250 300 350 400 450 500 250 300 350 400 450 500 Cutoff frequency (Hz) Cutoff frequency (Hz) (a) Case 0 (b) Case 2-T

1400 1400 2 2 1200 var(Y) = 898 Hz 1200 var(Y) = 3270 Hz C(Y) = 124 Hz C(Y) = 203 Hz 1000 1000 P(Y<300Hz) = 7.7% P(Y<300Hz) = 29.7% 800 800

600 600

400 400 Number of samples Number of samples

200 200

0 0 250 300 350 400 450 500 250 300 350 400 450 500 Cutoff frequency (Hz) Cutoff frequency (Hz) (c) Case 2-G (d) Case 2-BU

Figure 4-8: Histogram of Y resulting from various distributions for X2 considered in Case 2

represents a 50% reduction in C(X2) and a 75% increase in var (X2), and produces sizable increases the uncertainty estimates for the QOI.4 This illustrates that, as in

Case 1, the maximum entropy distribution for X2 given the available information (in this case, the uniform distribution since we are focusing on the interval [3.76, 5.64] µF) does not necessarily lead to the most conservative estimates for output uncertainty.

Figure 4-9 shows the entropy power-based sensitivity indices ηi and ζi alongside the main eﬀect indices Si computed from GSA. We see that for Cases 0, 2-T, and

2-G, Si is similar to ηi for both factors, and that ζi is equal to or slightly greater than

4Case 2-BU does not fall along the DSA curves shown in Figure 4-5(b). Since it entails a 75% increase in var (X2), one would need to extrapolate the uniform → bimodal uniform curve to 1 − δ = −0.75; doing so conﬁrms that a 60.7% increase in var (Y ) can reasonably be expected.

99 2.5 2.5 S S i i η η 2 i 2 i ζ ζ i i

1.5 1.5

1.11 1.14 1.17 1.13 1.02 1 0.98 1 0.80 0.72 0.67 0.59 0.5 0.5 0.33 0.27 0.20 0.18 0.00 0.00 0.00 0.00 0 0 Factor 1 Factor 2 Interactions Factor 1 Factor 2 Interactions

(a) Case 0 (b) Case 2-T

2.5 2.5 S S i i η η 2.06 2 i 2 i ζ ζ i i

1.5 1.5 1.18 1.19 1.00 1 1 0.85 0.87 0.75 0.57 0.57 0.5 0.43 0.5 0.35 0.38 0.13 0.13 0.00 0.00 0.00 0.00 0 0 Factor 1 Factor 2 Interactions Factor 1 Factor 2 Interactions

Figure 4-9: Main eﬀect and entropy power-based sensitivity indices for Case 2 one. This indicates that uncertainty apportionment using variance and entropy power produce similar results. Furthermore, it reveals that Gaussianity increases in moving from the input factors to the QOI. For Case 2-BU, however, ζ1 is less than one, ζ2 is exceedingly large, and η2 is signiﬁcantly smaller than S2. This implies that the proportion of output variance due to Factor 2 alone (87%) is dominated by the non- Gaussianity of the bimodal uniform distribution, with only a small contribution due to the intrinsic extent of X2. This is consistent with the result that the complexity of Y increases by only 11.9% over Case 0, whereas variance rises by a substantial 60.7% (see Table 4.2).

For Case 2-BU, our analysis has highlighted that the propagation of uncertainty from the inputs to the QOI is dominated by the bimodality of X2, which introduces

100 signiﬁcant non-Gaussianity into the system. Thus, in using entropy power decomposition, we learned more about the underlying features of the system that contribute to output uncertainty than can be revealed by variance-based sensitivity analysis alone. We conclude from Case 2 that initial assumptions regarding distribution shape can greatly impact uncertainty in the QOI, especially for systems involving multimodality or other highly non-Gaussian features. Therefore, one must be careful when selecting the appropriate distribution to characterize uncertainty in the system inputs, as mis- attributions can lead to gross over- or underestimates of uncertainty in the quantities of interest.

4.3 Design Budgets and Resource Allocation

Once the sources of uncertainty in a system are identified through sensitivity analysis, the task of uncertainty mitigation centers on making decisions regarding design modifications in the subsequent iteration [5, 6, 127]. Such decisions often relate to the allocation of resources in order to improve one or more aspects of the design. Resources can be allocated to a variety of different activities — for example, to direct future research, to conduct experiments, to improve physical and simulation-based modeling capabilities, or to invest in superior hardware or additional personnel. These steps complete the resource allocation feedback loop shown in Figure 1-2. Typically, the appropriate decisions for minimizing system uncertainty at each step of the design process are not known a priori. Instead, sensitivity analysis results are often used to infer the potential of various options for mitigating uncertainty. As discussed in Chapter 3, variance-based GSA and DSA, as well as relative entropy-based sensitivity analysis all enable the prioritization of factors according to contribution to output variability. However, these methods do not address how to identify or select between specific strategies. In the following sections, we use the R-C circuit example to demonstrate how the sensitivity analysis methods developed in this thesis can be combined with budgets for cost or uncertainty and used to uncover design alternatives for reducing uncer-

101 tainty, visualize tradeoﬀs among the available options, and ultimately guide decisions regarding the allocation of resources to improve system robustness and reliability.

4.3.1 Visualizing Tradeoﬀs

First, we are interested in studying how uncertainty estimates for the QOI are affected by reductions in the variance of one or both design parameters. For simplicity, we will assume that uncertainty in both resistance and capacitance can be well-characterized using uniform random variables. This assumption is justified by the DSA results in Figure 4-5, which showed that the variance of Y is not significantly affected by the shape of the underlying distributions for X1 and X2. Therefore, reductions in var (X1) 5 and var (X2) serve to shrink the extent of the corresponding uniform distributions. We note, however, that the techniques employed in the following analysis can be

adjusted to also accommodate changes in the distribution family of X1 or X2.

Figure 4-10 shows how different levels of reduction in var (X1) and var (X2) trade against one another with respect to variance, complexity, and risk remaining in the QOI. The bottom left corner of each figure represents the initial design (Case 0). The colored contours were generated using 10,000 MC simulations to evaluate the QOI at each design parameter combination. Hence, they represent the most accurate uncertainty estimates available for the system. For many realistic engineering systems, however, evaluating the QOI is computationally expensive, and it may not be feasible to perform numerous MC simulations in order to visualize tradeoffs in the design space. In those situations, it is still possible to approximate variance, complexity, and risk using sensitivity information. For variance, we can use the adjusted main effect sensitivity indices from DSA (i.e.,

Figure 4-5) to relate a 100(1 − δ)% decrease in var (X1) or var (X2) to the expected reduction in var (Y ). Similarly, we can combine the DSA results with local sensitivity estimates from (3.26) and (3.27) to predict the expected change in complexity and risk. The contours generated using these approximations are indicated by the dashed

5We assume that reductions in variance reﬂect improvements in component tolerance, and thus decrease distribution extent symmetrically about R = 100 Ω or C = 4.7 µF.

102 1 200 1 75 500 90 75 ) 200 ) 2 2 500 105 90 0.8 0.8 75 800 500 120 105 90 800 0.6 0.6 120 105 1100 135 800 1100 135 120 0.4 0.4150 1400 1100 1400 150 135 165 0.2 1700 0.2 1400 Percent reduction in var(X Percent reduction in var(X 150 1700 165 02000 0180 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 Percent reduction in var(X ) Percent reduction in var(X ) 1 1 (a) Lines of constant variance (b) Lines of constant complexity

1 0 0 0.025 ) 2 0.80.05 0.025 0 0.075 0.05 0.025 0.6 0.1 0.075 0.05 0.125 0.1 0.075 0.4 0.125 0.1 0.15 0.125 0.2 0.15 0.15 Percent reduction in var(X 0.175 0 0.175 0.175 0 0.2 0.4 0.6 0.8 1 Percent reduction in var(X ) 1 (c) Lines of constant risk

Figure 4-10: Contours for variance, complexity, and risk corresponding to reductions in factor variance, generated from 10,000 MC simulations (solid colored lines) or approximated using distributional or local sensitivity analysis results (dashed black lines) black lines in Figure 4-10. In Figure 4-10(a), there is a close match between the variance contours predicted from DSA results and those generated using MC simulations. For complexity and risk (Figures 4-10(b) and 4-10(c)), however, the correspondence deteriorates for large deviations from Case 0, especially as var (X1) is decreased. Nevertheless, the trends in variance, complexity, and risk resulting from reductions in var (X1) and var (X2) are correctly captured, which render the approximated contours useful for visualizing tradeoﬀs between uncertainty mitigation options in the input factors, while circumventing the need to perform additional MC simulations.

Figure 4-10 conﬁrms that decreasing var (X2) yields the most dramatic reductions

103 in output uncertainty; in fact, similar trends are observed for variance, complexity,

and risk. This is consistent with the prioritization of X2 over X1 in Section 4.2.1. Next, we will connect the potential for uncertainty reduction with the cost of implementing the requisite design change, which leads to a discussion of budgets for cost and uncertainty.

4.3.2 Cost and Uncertainty Budgets

In typical system design processes, uncertainty reduction efforts are constrained by the availability of resources, such as funds to acquire superior tools or components, manpower to conduct additional research or experiments, or time to allot to computational models. In this thesis, we assume that there is a cost associated with each activity that mitigiates uncertainty in the QOI. As an illustrative example, let us consider the notional curves shown in Figure 4-11(a) for the R-C circuit, which depicts 6 the cost associated with a 100(1 − δ)% reduction in the variance of X1 or X2. In Figure 4-11(b), the same information is visualized as contour lines of equal cost with respect to different proportions of variance reduction in the two factors. We observe that although reducing var (X2) is more effective in abating output uncertainty than decreasing var (X1), it is also more expensive to implement. Furthermore, the reduction in var (X2) is capped at 75% (that is, 25% of the variability in X2 is irreducible), whereas var (X1) can be reduced by as much as 99%. In order to make design decisions that adequately mitigate uncertainty yet are feasible to implement, we must take into account the available budgets, both in terms of cost and uncertainty. A cost budget specifies the amount of resources that can be expended to improve the design. An uncertainty budget, on the other hand, refers to the total level of uncertainty that is deemed tolerable for the system. In both cases, we seek to determine how much of the prescribed amount ought to be allocated to each design parameter. For the R-C circuit, we impose the following budgetary

6In general, it may be diﬃcult to determine the relationship between cost and reductions in input uncertainty. We assume in this thesis, however, that such relationships exist and have already been established using, for example, historical data, cost estimation methods, or expert opinion elicitation.

104 25 1 Factor 1 )

Factor 2 2 20 0.8 22.5

20 30 0.6 27.5 15 17.5 25 15 22.5 Cost 12.5 10 0.4 10 20 12.5 17.5 7.5 0.2 10 5 5 7.5 15 Percent reduction in var(X 5

0 0 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 Percent reduction in var(X ) 1−δ 1

(a) Cost of reducing variance of X1 or X2 (b) Lines of constant cost

Figure 4-11: Notional cost associated with a 100(1 − δ)% reduction in the variance of X1 or X2

constraints:

1. Complexity: The complexity with respect to the QOI shall not exceed 150 Hz.

2. Risk: The probability of violating the cutoﬀ frequency requirement shall not exceed 10%.

3. Cost: The cost of uncertainty reduction shall not exceed 20 units of cost.

Next, we examine each constraint individually in turn to study the resulting optimal allocation. Figure 4-12 overlays contours of variance, complexity, and risk in the QOI (solid colored lines) with those corresponding to cost of implementation. These contours allow us to consider the cost and uncertainty budgets in conjunction. We see that for a given cost, the maximum possible uncertainty reduction is achieved

by decreasing the variance of both X1 and X2 simultaneously, rather than focusing on either factor alone. Table 4.3 lists the lowest possible uncertainty and cost estimates for the system when each of the budgetary constraints (C(Y ) ≤ 150 Hz, P (Y < 300 Hz) ≤ 10%, Cost ≤ 20) is in turn made active. Of the three budgets, the complexity constraint is the cheapest to satisfy; however, the resulting allocation of factor variance reduction does not satisfy the risk constraint. To ensure that risk does not exceed 10%, a minimum cost of 14.8 is required; this corresponds to a complexity of 132 Hz. Finally, a cost budget of 20 is more than adequate to guarantee that the

105 1 200 1 75 500 90 75 ) 200 ) 2 2 500 105 90 0.8 0.8 75 800 500 120 105 90 20 30 20 30 800 20 20 0.6 0.6 120 105 1100 25 135 25 15 800 15 110015 135 15 120 0.4 0.4150 1400 201100 20 10 10 1400 150 135 165 0.2 1700 15 0.2 15 5 10 1400 5 Percent reduction in var(X 5 Percent reduction in var(X 5 10 150 1700 165 02000 0180 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 Percent reduction in var(X ) Percent reduction in var(X ) 1 1 (a) Lines of constant variance and cost (b) Lines of constant complexity and cost

1 0 0 0.025 ) 2 0.80.05 0.025 0 0.075 0.05 20 0.02530 0.6 0.1 0.07520 250.05 15 0.1 0.125 15 0.075 0.4 0.125 200.1 10 0.15 0.125 0.2 15 5 0.15 0.15

Percent reduction in var(X 5 10 0.175 0 0.175 0.175 0 0.2 0.4 0.6 0.8 1 Percent reduction in var(X ) 1 (c) Lines of constant risk and cost

Figure 4-12: Contours for variance, complexity, and risk corresponding to reductions in factor variance (solid colored lines) overlaid with contours for cost of implementation (dashed green lines)

uncertainty budgets are also satisﬁed. As we observe in Figures 4-12(b) and 4-12(c),

the expenditure of 20 units of cost is sufficient to decrease var (X2) by up to 67%; even without a simultaneous reduction in var (X1), it is enough to reduce complexity and risk to acceptable levels of 125 Hz and 8%, respectively. The results in Table 4.3 suggest that the cost and uncertainty budgets for the R-C circuit design are mutually compatible and achievable. There are a host of solutions that satisfy all three budgetary constraints, which lie within the shaded region in Figure 4-13. The main tradeoff in these solutions is between cost and risk: when both of those constraints are satisfied, the complexity budget is automatically satisfied as well. In the following section, however, we will consider the cost and

106 Active constraint var (Y ) C(Y ) P (Y < 300 Hz) Cost Complexity 1378 150 13.4% 8.8 Risk 1055 132 10% 14.8 Cost 780 114 6.4% 20

Table 4.3: Best achievable uncertainty mitigation results given individual budgets for cost and uncertainty. Entries in red denote budget violations.

Figure 4-13: Contours for risk (solid blue lines) overlaid with contours for cost (dashed green lines). Shaded region denotes area in which all budget constraints are satisﬁed. uncertainty budgets alongside available design options, which can lead to diﬀerent conclusions regarding their feasbility.

4.3.3 Identifying and Evaluating Design Alternatives

In this section, we study how the nominal value and tolerance of the components in the R-C circuit trade against one another in terms of contribution to uncertainty in the QOI. The present analysis builds upon the design budget discussion from Section 4.3.2, and enables designers to visualize strategies for uncertainty mitigation in terms of actionable items in the design space.

For each of X1 and X2, there are two parameters that can be modiﬁed: the nominal value of the circuit component (R or C) and the tolerance (Rtol or Ctol). We vary

R and C from their nominal values by up to ±20%, and decrease Rtol and Ctol from the initial tolerances of ±10% and ±20%, respectively, down to ±1%. Figures 4-14

107

6 6 0.3 0.5 0.7 0.9 180 150 0.1 5.5 5.5

0.01 0.7 F) 150 F) 0.3 0.5 µ 5 210 180 µ 5 0.1 4.5 4.5 0.01 0.3 240 210 180 4 4 0.1 Capacitance ( Capacitance ( 270 0.01 240 3.5 Case210 0 3.5 Case 0 Option A Option A 300 Option B Option B 330 270 240 3 3 80 90 100 110 120 80 90 100 110 120 Resistance (Ω) Resistance (Ω) (a) Lines of constant complexity (b) Lines of constant risk

Figure 4-14: Uncertainty contours for variations in resistance and capacitance, generated from 10,000 MC simulations (solid colored lines) or approximated using entropy power-based sensitivity analysis results (dashed black lines)

0.01 0.1 0.1 0.1 0.2 0.4 0.5 0.6

200 0.3 220220 210 0.08 210 0.08

190 180 170 160 150

0.06 140 0.06

200 0.01 0.04 0.04 0.4 0.6 200 190 180 170 160 150 0.1 0.2 0.3 0.5 Resistor tolerance Resistor tolerance

190 180 170 160 150 0.02 Case 0 140 0.02 Case 0

130 140 Option A Option A Option C Option C 0 0 80 90 100 110 120 80 90 100 110 120 Resistance (Ω) Resistance (Ω) (a) Lines of constant complexity (b) Lines of constant risk

Figure 4-15: Uncertainty contours for variations in resistance and resistor tolerance through 4-17 show contour lines for complexity (left) and risk (right) for different combinations of the four quantities; a black X in each figure denotes the location of Case 0 in the design space. These contours reveal several interesting trends. First, we see that raising either R or C lowers complexity in the QOI but also increases risk, and vice-versa (Figure 4-14); this indicates the presence of competing objectives. For a fixed value of R, decreasing Rtol appears to have little effect on complexity, and almost no effect on risk (Figure 4-15). Figure 4-16(a), on the other hand, shows that both C and Ctol

108

0.2 0.2 0.01 0.1 0.2 0.3 0.4 200

260 0.5 240 180 0.6 220 160 140

0.15 0.15 120

0.7 0.8 200 180 160 140 0.1 0.1

0.01

0.1 0.2 0.3 120 100 0.4

0.6 80 0.5 140

Capacitor tolerance 0.05 Capacitor tolerance 0.05 Case 0 Case 0 100 120 Option B Option B 0.9

0.7 80 Option D Option D 0.8 0 0 3 3.5 4 4.5 5 5.5 6 3.5 4 4.5 5 5.5 6 Capacitance (µF) Capacitance (µF) (a) Lines of constant complexity (b) Lines of constant risk

Figure 4-16: Uncertainty contours for variations in capacitance and capacitor tolerance

0.2 160 0.2 0.16 0.16 0.16 140 0.13 160 0.13 0.1 0.13 140 0.1 0.1 0.15 120 0.15 0.07 0.04 120 0.07 0.07 100 0.01 0.04 0.04 0.1 80 100 0.1 0.01 80 0.04 60 0.01 0.01

Capacitor tolerance 0.05 Capacitor tolerance 0.05 Case 0 Case 0 40 60 80 Option C Option C 20 Option D Option D 0 0 0 0.02 0.04 0.06 0.08 0.1 0 0.02 0.04 0.06 0.08 0.1 Resistor tolerance Resistor tolerance (a) Lines of constant complexity (b) Lines of constant risk

Figure 4-17: Uncertainty contours for variations in resistor and capacitor tolerance

are inﬂuential with respect to complexity; however, Figure 4-16(b) indicates that 7 depending on the value of C, decreasing Ctol can either increase or decrease risk.

Finally, Figure 4-17 shows that when trading Rtol against Ctol, decreasing the latter is more eﬀective for reducing both complexity and risk. As was the case in Section 4.3.1, the colored contours in Figures 4-14 through 4-17 were generated using 10,000 MC simulations to evaluate Y at each design parameter combination. The dashed black lines correspond to local approximations obtained

7This is because changing C also shifts the distribution of Y relative to the requirement r = 300 Hz (see Figure 3-5).

109 using sensitivity information, and can be produced without performing additional MC simulations. The approximated risk contours in Figures 4-14(b), 4-15(b) 4-16(b), and 4-17(b) were computed using local sensitivity results from (3.33) and (3.37). Although these approximations vary in accuracy, the overall trends in risk due to changing R,

C, Rtol, and Ctol are captured, especially for design parameter combinations close to Case 0. The approximated complexity contours (Figures 4-14(a), 4-15(a) 4-16(a), and 4-17(a)), on the other hand, rely on the results of entropy power-based sensitivity analysis, and were estimated using the procedure below.

o o First, we introduce the notation X1 and X2 , corresponding to the random variables 0 0 for resistance and capacitance in Case 0, respectively, as well as X1 and X2, denoting the updated estimates for those factors after some design modiﬁcaton. Similarly, letting i = 1, 2, we write the following relations:

o o 0 0 Zi = gi(Xi ) Zi = gi(Xi)

o o o 0 0 0 Y = g(X1 ,X2 ) Y = g(X1,X2) o oG 0 0G o exp[2DKL(Zi ||Zi )] 0 exp[2DKL(Zi||Zi )] ζi = o oG ζi = 0 0G exp[2DKL(Y ||Y )] exp[2DKL(Y ||Y )]

Based on the entropy power decomposition given in (3.44), we can express the entropy power of Y o as the sum of the following terms:

o o o o o o o N(Y ) = N(Z1 )ζ1 + N(Z2 )ζ2 + N(Z12)ζ12. (4.6)

Since the interaction eﬀects are negligible for this system, we can drop the last term in (4.6). Then, we employ the following approximation to predict N(Y 0), which uses o o the non-Gaussianity indices ζ1 and ζ2 evaluated in Case 0:

0 0 o 0 o N(Y ) ≈ N(Z1)ζ1 + N(Z2)ζ2 . (4.7)

The relationship between each input factor Xi and the corresponding auxiliary ran-

dom variable Zi is available from the ANOVA-HDMR (computed either analytically

110 or numerically); for the R-C circuit, these relationships are given in (4.2)–(4.5). Thus, 0 0 0 0 we are able to calculate N(Z1) and N(Z2) based on X1 and X2, which then allows us to estimate N(Y 0) without having to explicitly evaluate Y 0 using MC simulations. Finally, recognizing that entropy power is proportional to the square of complexity, we can rewrite (4.7) in terms of C(Y 0):

0 p 0 o 0 o C(Y ) ≈ 2πe[N(Z1)ζ1 + N(Z2)ζ2 ]. (4.8)

We observe that the approximations obtained using (4.8) closely match the colored contours generated from brute force MC simulation. The largest discrepancies occur in Figure 4-15(a) for small resistor tolerances, which is consistent with the trend

observed in Figures 4-10(b) and 4-10(c) for small values of var (X1). Even for this case,

however, the overall trends in complexity due to varying Rtol and R are preserved. Thus, the approximation given in (4.8) provides an easy way to visualize trends in complexity and identify regions in the design space that warrant additional analysis — toward which additional MC simulations can be directed to generate more rigorous estimates of uncertainty.

Figures 4-14 through 4-17 depict tradeoﬀs among R, C, Rtol, and Ctol as continuous within the design space, implying inﬁnitely many design combinations. For many engineering systems, however, the design space instead contains a discrete number of feasible options dictated by component availability. Thus, we next investigate how local sensitivity analysis results can be used to identify distinct design alternatives.

Using (3.38) and (3.39), we can compute the requisite change in the mean and standard deviation of X1 and X2 needed to achieve a desired decrease in risk. In this case, we set ∆P = −0.086 so as to meet the constraint that P (Y < 300 Hz) ≤ 10%. Table 4.4 lists the change in mean and standard deviation of each factor required to achieve this risk reduction as predicted using local sensitivities, as well as the corresponding component nominal value and tolerance. Consistent with the risk contours in Figure 4-14(b), we see that reducing risk requires decreasing R or C. This suggests that we use a resistor with a nominal value of no more than 96.6 Ω, or a

111 Change mean only Change SD only Component

∆µXi Nominal value ∆σXi Tolerance Resistor −3.4 Ω R = 96.6 Ω −8.4 Ω N/A

Capacitor −0.16 µF C = 4.54 µF −0.17 µF Ctol = ±14.4%

Table 4.4: Local sensitivity predictions for the change in mean and standard deviation (SD) of each factor required to reduce risk to 10%

capacitor with a nominal value of no more than 4.54 µF. Alternatively, we can reduce risk by using components with tighter tolerances. For the resistor, the change in the standard deviation of X1 needed to satisfy the risk constraint is −8.4 Ω; however, in

Case 0, σX1 = 5.8 Ω, which implies that it is not possible to achieve the requisite risk

reduction by decreasing Rtol alone. This is consistent with the risk contours in Figures 4-15(b) and 4-17(b). For the capacitor, we observe that the risk constraint can be

met by decreasing the standard deviation of X2 by −0.17 µF, which corresponds to a component tolerance of ±14.4%. The local sensitivity analysis predictions point the designer to values of R, C,

Rtol, and Ctol that would satisfy the uncertainty budget for risk. Consulting standard oﬀ-the-shelf resistor and capacitor values [128], we identify four design alteratives — Options A–D — which correspond most closely to the desired component speci- ﬁcations shown in Table 4.4. Options A–D are listed in Table 4.5 along with their associated estimates of variance, complexity, risk, and cost of implementation. Each

option perturbs one of R, C, Rtol, and Ctol from its initial value in Case 0. We assume in this analysis that it is comparatively cheaper to purchase components with different nominal values (Options A and B) than it is to acquire components with significantly tighter tolerances (Options C and D).8 The locations of the four options within the design space are indicated in Figures 4-14 through 4-17. Figure 4-18 illustrates the tradeoffs between complexity and risk for the various

8 Assuming that X1 and X2 are both uniform random variables, a decrease in Rtol from ±10% in Case 0 to ±1% in Option C corresponds to a 99% reduction in var (X1), and a decrease in Ctol from ±20% in Case 0 to ±10% in Option D corresponds to a 75% reduction in var (X2). The associated costs of implementation correspond to the rightmost red and blue points in Figure 4-11(a), respectively.

112 Option R (Ω) Rtol C (µF) Ctol var (Y ) C(Y ) P (Y < 300 Hz) Cost Case 0 100 ±10% 4.7 ±20% 2015 181 18.6% N/A A 82 ±10% 4.7 ±20% 2997 221 0% 5 B 100 ±10% 3.9 ±20% 2927 218 0% 5 C 100 ±1% 4.7 ±20% 1627 154 18.1% 13.2 D 100 ±10% 4.7 ±10% 788 115 7.1% 22.5

Table 4.5: Uncertainty and cost estimates associated with various design options. Entries in red denote budget violations. options. Immediately, we see that none of the options satisﬁes both the cost and uncertainty budgets. Options A and B fall well within the cost budget and reduce risk to zero; however, they both increase complexity to unacceptable levels, and thus can be eliminated from consideration.9 In Option C, the resistor tolerance is reduced to just ±1% at a cost of 13.2; yet, there is virtually no change in risk compared to Case 0. Option D appears to be the most promising choice, as it is the only one to satisfy both the complexity and risk constraints. However, it exceeds the cost budget of 20.

0.25 Case 0 Option A 0.2 Option B Option C Option D 0.15 Risk 0.1

0.05

0 0 50 100 150 200 250 Complexity (Hz)

Figure 4-18: Tradeoﬀs in complexity and risk for various design options. Dashed green lines bound region where both complexity and risk constraints are satisﬁed.

9Had we instead identiﬁed options that raised the nominal values of R and C, the result would have decreased complexity and increased risk, which is similarly undesirable.

113 These results suggest that in order to meet both the cost and uncertainty budgets, the designer must either seek alternative options, or relax one or more of the constraints. In Figure 4-18, we see that Option D falls well within the region of acceptable uncertainty. Focusing on the top right corner of that region, we ﬁnd that when

Ctol = ±12.5%, the system achieves C(Y ) = 130 Hz and P (Y < 300 Hz) = 10.0% at a cost of 18.3, which meets all design budgets. However, while this design is mathematically possible, it is not realistically feasible given limitations in component availability, as standard capacitor tolerances include ±20%, ±10%, ±5%, ±2%, and ±1%. From this analysis, we conclude that the most promising course of action is to allocate resources toward improving capacitor tolerance; furthermore, we recommend increasing the cost budget so that all uncertainty targets can be satisﬁed using standard components.

4.4 Chapter Summary

In this chapter, we demonstrated the uncertainty quantification and sensitivity analysis methods introduced in Chapters 2–3 on a simple R-C high-pass filter circuit. This example highlighted the applicability of our methods to assess the implications of initial assumptions about the system for uncertainty in the quantities of interest. We also showed how various sensitivity results can be used to formulate local approximations of output uncertainty, which enable the visualization of tradeoffs in the design space without expending additional computational resources to perform model evaluations. Finally, we connected our sensitivity analysis techniques to budgets for cost and uncertainty, which facilitates the identification and evaluation of various design alternatives and helps inform decisions for allocating resources to mitigate uncertainty. In Chapter 5, we will apply the methods developed in this thesis to perform cost- benefit analysis of a real-world aviation environmental policy, further demonstrating their utility for improving system robustness and reliability.

114 Chapter 5

Application to a Real-World Aviation Policy Analysis

This chapter presents a case study in which the methods developed in this thesis are applied to analyze a set of real-world aviation policies pertaining to increases in aircraft noise certification stringency. Section 5.1 provides some background on aviation noise impacts and frames the objectives of the case study. Section 5.2 introduces the APMT-Impacts Noise Module and explains how it is used in the present context to perform a cost-benefit analysis of the proposed policy options, identify sources of uncertainty, and analyze assumptions and tradeoffs. Section 5.3 discusses the case study results and recommends specific courses of action to support the design of cost-beneficial policy options.

5.1 Background and Problem Overview

5.1.1 Aviation Noise Impacts

The demand for commercial aviation has risen steadily over the last several decades, and is expected to continue growing at a rate of 5% per year over at least the next two decades [129–131]. This wave of growth brings with it increasing concerns regarding the environmental impacts of aviation, which include aircraft noise, air quality degra-

115 dation, and climate change. Of these issues, aircraft noise has the most immediate and perceivable community impact [132–134], and was the first to be regulated by the International Civil Aviation Organization (ICAO) in 1971 with the publication of Annex 16: Environmental Protection, Volume I: International Noise Standards [135]. There are many effects associated with exposure to aircraft noise, which can be broadly classified as either physical or monetary. The physical effects of aviation noise span a range of severities, and include annoyance, sleep disturbance, interference with school learning and work performance, and health risks such as hypertension and heart disease [136, 137]. The monetary effects, on the other hand, include housing value depreciation, rental loss, and the cost associated with lost work or school performance. The assessment of these monetary impacts is of particular interest to policymakers, researchers, and aircraft manufacturers, as they provide quantitative measures with which to perform cost-benefit analysis of various policy options.1 Motivated by a 2004 report to the US Congress on aviation and the environment [138], the Federal Aviation Administration (FAA) is developing a comprehensive suite of software tools that can characterize and quantify a wide spectrum of environmental effects and tradeoffs, including interdependencies among aviation-related noise and emissions, impacts on health and welfare, and industry and consumer costs under various scenarios [139]. This effort, known as the Aviation Environmental Tools Suite, takes as inputs proposed aviation policies or scenarios of interest, which can pertain to regulations (e.g., noise and emissions stringencies, changes to aircraft operations and procedures), finances (e.g., fees or taxes), or anticipated technological improvements. These inputs are processed through various modules of the Tools Suite — in particular the Aviation environmental Portfolio Management Tool for Impacts (APMT-Impacts) — to produce cost-benefit analyses that explicitly detail the monetized noise, climate, and air quality impacts of the proposed policy or scenario with respect to a well- defined baseline [140].

1While the physical and monetary effects of aviation noise are usually presented separately, the two categories are not necessarily independent – that is, monetary impacts (typically assessed from real estate-related damages) often serve as surrogate measures for the wider range of interrelated effects that are difficult to quantify individually. However, it is recognized that such estimates may undervalue the full impacts of noise.

116 The Noise Module within APMT-Impacts was developed by He et al. and relates city-level per capita income to an individual’s willingness to pay for one decibel (dB) of noise reduction [141, 142]. It is based on a meta-analysis of 63 hedonic pricing aviation noise studies, which used observed diﬀerences in housing markets between noisy and quiet areas to determine the implicit value of quietness (or conversely, the cost of noise).2 Because income data are typically readily available for most airport regions around the world, the income-based model facilitates the assessment of aviation noise impacts on a global scale. For example, a previous study used the APMT-Impacts Noise Module to estimate that capitalized property value depreciation due to aviation noise in 2005 totaled $23.8 billion around 181 airports worldwide, including $9.8 billion around 95 airports in the US [142]. In the following sections, we will discuss the APMT-Impacts Noise Module in more detail, as well as introduce the aviation policy case study to which it is applied.

5.1.2 CAEP/9 Noise Stringency Options

Since ICAO’s initial publication of Annex 16 in 1971, international aircraft noise certification standards have been updated on three different occasions to reflect ad- vances in technology. The most recent stringency increase was adopted in 2001 as part of the ICAO Committee for Aviation Environmental Protection’s (ICAO-CAEP) CAEP/5 cycle. In 2007, a formal review of existing noise certification standards was initiated as part of CAEP/7. Subsequently, five options for noise stringency increase were proposed during CAEP/9 in 2013, which correspond to changes in noise certification stringency of −3 (Stringency 1), −5 (Stringency 2), −7 (Stringency 3), −9 (Stringency 4), and −11 (Stringency 5) cumulative EPNdB relative to current limits

2Hedonic pricing (HP) studies typically derive a Noise Depreciation Index (NDI) for one airport, which represents the percentage decrease in property value corresponding to a one decibel increase in the noise level in the region. Previous meta-studies have shown that NDI values tend to be similar across countries and stable over time [143–145]. An alternative to HP is contingent valuation (CV), which uses survey methods to explicitly determine individuals’ willingness to pay for noise abatement, or alternatively, willingness to accept compensation for noise increases. While HP and CV methods both have respective advantages, some key drawbacks which limit their applicability to cost-benefit analysis of aviation policies include insufficient availability of real-estate data (HP) [146], inconsistency of results between different study sites (CV) [147, 148], and prohibitive cost of performing new valuation studies (both).

117 [149].3

The CAEP/9 noise stringency goals are achieved through a mixture of changes to ﬂeet composition and aircraft operation projected by the ICAO Forecasting and Economic Support Group (FESG). For the purpose of policy cost-beneﬁt analysis, the noise impacts of the proposed stringency options are modeled for 99 airports in the US using the FAA’s Aviation Environmental Design Tool (AEDT) [152]. These 99 airports are listed in Appendix D and belong to the FAA’s database of 185 Shell-1 airports worldwide that account for an estimated 91% of total global noise exposure [153]. The noise contours for each airport are expressed in day-night average sound level (DNL) in increments of 5 dB: 55–60 dB, 60–65 dB, 65–70 dB, 70–75 dB, and 75 dB and above.4 The noise contours corresponding to each stringency option, along with a baseline scenario with no stringency change, are forecasted from 2006 (the reference year of aviation activity) to 2036, with a policy implementation year of 2020.

Along with forecasts of future noise, each CAEP/9 noise stringency option also has an associated cost of implementation, which is the sum of the recurring and non- recurring costs estimated by the FESG.5 Recurring costs are incurred annually and consist of fuel costs, capital costs, and other direct operating costs, and were cal- culated using the Aviation environmental Portfolio Management Tool for Economics (APMT-Economics) under the assumptions documented in Ref. [149]. Non-recurring costs include loss in ﬂeet value and cost to manufacturers, and are provided as low and high estimates for each stringency option.

A policy is said to be cost-beneficial (resulting in a “net benefit”) if its associated environmental benefits (typically negative) exceed the implementation costs (typically

3The eﬀective perceived noise in decibels (EPNdB) is the standard metric used in aircraft noise certiﬁcation limits, and represents the weighted sum of the sound pressure level from nine contiguous frequency ranges, with added penalties for discrete pure tones and duration of noise [150, 151]. 4The day-night average sound level (DNL) is the 24-hour A-weighted equivalent noise level with a 10 dB penalty applied for nighttime hours. A similar measure, the day-evening-night average sound level (DENL), is commonly used in Europe; DENL is very similar to DNL, except that it also applies a 5 dB penalty to noise events during evening hours. 5Recurring and non-recurring costs were estimated on a global scale. To obtain US-only policy implementation costs, it was assumed that US operations comprise approximately 27% of the global sum, based on previous analyses conducted using APMT-Economics [154].

118 positive), producing a negative value for net monetary impacts.6 Conversely, a policy is not cost-beneﬁcial (resulting in a “net dis-beneﬁt”) if the net monetary impact is a positive value.

The objectives of this case study are to explicitly model uncertainties in the CAEP/9 noise stringency options and the APMT-Impacts Noise Module, understand their implications for policy net cost-benefit, and identify resource allocation strategies for improving robustness and reliability. The outcomes of this study can help inform decisions to support the design of cost-beneficial policy implementation responses. Prior work by Wolfe and Waitz used various modules within APMT-Impacts to estimate the net present value of each stringency option with respect to noise, climate, and air quality impacts [155]. The cost of policy implementation was treated as a deterministic value equal to the low estimate for each stringency and discount rate. It was found that Stringencies 4 and 5 resulted in a net dis-benefit across all discount rates considered, even though the lowest (least conservative) cost estimates were used. In this case study we will use random variables to characterize uncertainty in the implementation cost, and focus only on CAEP/9 Stringencies 1–3. Further- more, we will limit the scope of environmental effects to consider only noise impacts. Under these assumptions, it is expected that the present analysis will provide a more conservative estimate of policy net cost-benefit for Stringencies 1–3.

5.2 APMT-Impacts Noise Module

The APMT-Impacts Noise Module is a suite of scripts and functions implemented in the MATLAB R (R2013a, The MathWorks, Natick, MA) numerical computing environment. The next sections describe the inputs, assumptions, and impacts estimation procedure used in the module.

6We note, however, that depending on the policy, environmental impacts can be either positive or negative; the same is true of implementation costs.

119 5.2.1 Model Inputs

There are a number of inputs to the APMT-Impacts Noise Module, which can be classiﬁed into three broad categories: scenario, scientiﬁc, and valuation. Below, we describe each category and its respective inputs in greater detail.

Scenario Inputs

The scenario inputs correspond to the policy under consideration, and include de- mographic data for population and personal income, alternative forecasts of future aviation activities or situations, and the cost of implementing the proposed policy. As discussed in Section 5.1.2, noise contours from AEDT correspond to the DNL of aircraft noise at a particular location, and are computed as yearly averages around each airport. To evaluate the monetary impacts of a particular aviation policy, two sets of noise contours are needed: baseline and policy. The baseline contours for the reference year are constructed using actual aircraft movement data from a represen- tative day of operations. The baseline forecast for future years represents an estimate of the most likely future noise scenario while maintaining the status quo for technology, fleet mix, and aviation demand. The policy forecast reflects the expected future noise levels after the implementation of the proposed policy. The monetary impacts due to policy implementation correspond to the difference between the policy and the baseline scenarios (henceforth termed the “policy-minus-baseline” scenario). As will be further explained in Section 5.2.2, the APMT-Impacts Noise Module assesses monetary noise impacts based on a per person willingness to pay (WTP) for one decibel of noise reduction, which is computed as a function of the average city-level personal income. Thus, locality-specific population and income data are required. Income data may be obtained from a variety of sources, such as national or local statistical agencies, or the US Bureau of Economic Analysis, which publishes the annual personal income for each Metropolitan Statistical Area (MSA) [156]. The income level for each of the 99 airports in the CAEP/9 analysis is listed in Appendix D. Similarly, population data can be obtained from several sources, including the decadal US Census, the European Environmental Agency’s population maps, the

120 Gridded Rural-Urban Mapping Project, and national or local statistical agencies. For the CAEP/9 noise stringency analysis, noise and population data are provided jointly in the form of the number of persons residing within each 5 dB contour band surrounding an airport in the years 2006, 2026, and 2036. The population data were derived from the 2000 US Census, and thus population changes since that time are not accounted for. Finally, under the current capabilities of the APMT-Impacts Noise Module, uncertainty associated with income and population data are not modeled. The cost of policy implementation (IC) is another example of a scenario input. In the CAEP/9 analysis, the estimated total cost for a particular stringency option can be obtained by summing the recurring and non-recurring costs for a selected discount rate. There are two cost totals for each stringency — low and high — corresponding to the best- and worst-case estimates, respectively. To characterize uncertainty in the implementation cost, we model this input using a uniform random variable on the interval bounded by the low and high estimates.

Scientiﬁc Inputs

The scientific inputs enable the estimation of monetary impacts for a scenario of interest, and consist of the background noise level (BNL), noise contour uncertainty (CU), and the regression parameters (RP) that relate personal income to the WTP for noise abatement. The uncertainties associated with these inputs arise from limitations in scientific knowledge or modeling capabilities and are represented probabilistically using random variables. Background noise level refers to the level of ambient noise in an airport region, which is important since monetary impacts are assessed only when the level of aviation noise exceeds this threshold. Although BNL can vary from region to region, typical values range between 50–60 dB DNL [143, 148]. To capture uncertainty in this input, we model BNL as a symmetric triangular distribution between 50 dB and 55 dB, with a mode of 52.5 dB. Under the current capabilities of AEDT, the computed noise contours are fixed values. In order to account for uncertainty in the noise contour level, we assume

121 a triangular distribution for CU with minimum, maximum, and mode at −2 dB, 2 dB, and 0 dB, respectively. This distribution represents an engineering estimate and captures only uncertainty in the noise contour level, not in the area of the contour, which can also aﬀect the estimated monetary noise impacts [157].

1000

500

−500 Intercept ($/dB/person)

Mean = 39.11 SD = 159.65 −1000 Income coefficient (1/dB/person) −0.02 0 0.02 0.04 0.06 0 2000 1500 1000 500 0 Number of samples

500

1000 Mean = 0.0114 SD = 0.0061 1500 Number of samples

2000

Figure 5-1: Joint and marginal distributions for the regression parameters obtained using 10,000 bootstrapped samples

Finally, the regression parameters consist of two quantities, income coeﬃcient and intercept, which specify a linear relationship between income and WTP for 1 dB of reduction in aviation noise. For US airports, this relationship is given by:

WTP = Income coeﬃcient × Income + Intercept. (5.1)

122 Since income coeﬃcient and intercept are not independent, they are modeled jointly as a random variable in R2. The joint distribution and corresponding marginal distributions are shown in Figure 5-1.7 These distributions are approximately Gaussian in shape, and were obtained by bootstrapping the 63 meta-analysis observations of WTP versus per capita income and performing weighted least-squares regression using a robust bisquare estimator [142].

Valuation Inputs

Finally, the valuation inputs pertain to subjective judgments regarding the value of money over time or the significance of environmental impacts, rather than quantities rooted in scientific knowledge. These factors are modeled as deterministic values instead of random variables. The valuation inputs of the APMT-Impacts Noise Module consist of the discount rate, income growth rate, and significance level. The discount rate (DR) captures the depreciation in the value of money over time, and is expressed as an annual percentage. It is closely related to the time span of the analyzed policy, which is based on the typical economic life of a building and the duration of future noise impacts that is considered by the house buyers. In this analysis, the policy time span is 30 years (2006–2036). Consistent with previous analyses of the CAEP/9 noise stringency options, the nominal discount rate is selected to be 3% [155]. However, other discount rates (5%, 7%, 9%) will also be considered to investigate the effect of this parameter on net cost-benefit. We assume that the same discount rate is used to compute the net present value of both monetary noise impacts and recurring implementation costs. The income growth rate (IGR) represents the annual rate of change in the city- level average personal income. It is universally applied to the income levels of all

7In the case of non-US airports, a third regression parameter, the interaction term, is added to (5.1), such that WTP is given by:

WTP = (Income coeﬃcient + Interaction term) × Income + Intercept. (5.2)

The interaction term has an approximately Gaussian marginal distribution with mean and standard deviation equal to 39.11 and 159.65, respectively [142]. For this US-based case study, however, we will not further discuss the interaction term.

123 airports in the analysis when calculating the WTP for noise abatement. While this parameter may be user-selected to be any reasonable value (even negative growth rates), in this analysis it is set to zero so as to consider noise impacts solely due to the growth of aviation, rather than due to changes in economic activity. The significance level (SL) is the threshold DNL above which aircraft noise is considered to have significant impact on the surrounding community. It does not affect the value of the computed monetary noise impacts, but rather designates impacts as significant or insignificant, and thereby includes or excludes them from the reported results. In this analysis, the significance level is set to 52.5 dB, as consistent with the midrange lens definition in the APMT-Impacts Noise Module.8 Noting that the lowest noise contour level in the CAEP/9 analysis is 55 dB, this choice implies that any aviation noise above the background noise level is perceived as having a significant impact on the community.

Symbol Factor (Units) Definition BNL Background noise level (dB) T (50, 55, 52.5) CU Contour uncertainty (dB) T (−2, 2, 0) Income coefficient (1/dB/person) Mean = 0.0114, SD = 0.0061 RP Intercept ($/dB/person) Mean = 39.11, SD = 159.65 SL Significance level (dB) 52.5 dB IGR Income growth rate (%) 0% IC Implementation cost ($) U [low estimate, high estimate] DR Discount rate (%) 3%, 5%, 7%, 9%

Table 5.1: Inputs to CAEP/9 noise stringency case study. The deﬁnitions for BNL, CU, RP, SL, and IGR correspond to the midrange lens of the APMT-Impacts Noise Module, whereas IC and DR are used in the post-processing of impacts estimates to compute net cost-beneﬁt.

Table 5.1 summarizes the main model inputs presented in this section. We note that with the exception of implementation cost, which is added directly to the net present value of monetary noise impacts (to be discussed in Section 5.2.2), all inputs

8A lens is a ready-made set of input values that can be used to evaluate policy options given a particular perspective or outlook. The midrange lens represents a most likely scenario, where all model inputs are set to their nominal value or distribution.

124 listed in the table are used in impacts estimation for both the baseline and policy scenarios. Thus, while uncertainties in these inputs affect the baseline or policy scenario results to the first-order, when considering a policy-minus-baseline scenario, their effects become second-order. Finally, for each of the probabilistic inputs (BNL, CU, RP, and IC), 10,000 MC samples are used to characterize the distribution.9

5.2.2 Estimating Monetary Impacts

In this case study, we use the APMT-Impacts Noise Module to assess the net cost- beneﬁt of each CAEP/9 noise stringency option, which we estimate as the net present value of the sum of the monetary noise impacts and implementation cost. A schematic of the module is shown in Figure 5-2. For the kth airport in the analysis, the capitalized monetary impacts of aviation in year t in the dth 5 dB noise contour band is given by V (k, d, t):

V (k, d, t) = WTP(k, t) × ∆dB(k, d, t) × Number of persons(k, d, t), (5.3) where WTP(k, t) is obtained using (5.1), and ∆dB(k, d, t), the aviation noise level above the BNL, is computed as:

∆dB(k, d, t) = Noise contour level(k, d, t) + CU − BNL. (5.4)

To estimate the total capitalized monetary impacts in year t, we sum V (k, d, t) over all noise contour bands and across all airports:

X X V (t) = V (k, d, t). (5.5) k d

It is important to note that V (t) is a capitalized value that encompasses not only the property value depreciation due to aviation noise in year t, but also the future

9In order to check for convergence of the MC samples, the running mean and variance of each input were plotted versus the sample number. After 10,000 samples, ﬂuctuations in the running mean and variance were on the order of 0.1% for all inputs.

125 Model Inputs Intermediate Quantities of Scientific & Results Interest Scenario

Valuation PhysicalImpacts Contour Exposure Noise Uncertainty Area Contours (dB) (m2) (dB) AEDT Background Exposed Exposed Noise Level Population Population

(dB) (Persons) (Persons)

Population Data (Persons/m2) Significance ΔdB US Census, Level (dB) EEA, GRUMP (dB)

Personal Income Willingness Capitalized Income Growth Rate to Pay Impacts ($) (%) ($/Person/dB) ($) MonetaryImpacts USBEA, Statistical Agencies Capital Annual Regression Recovery Impacts Parameters Factor ($) Implemen- tation Cost

($) Discount Implemen- Net Present APMT- Rate tation Cost Value Economics (%) ($) ($)

Figure 5-2: Schematic of APMT-Impacts Noise Module [142, adapted from Figure 4]

noise damages anticipated by the house buyers. This quantity can be transformed into a net present value (NPV) by applying a selected discount rate:

T X V (t) − V (t − 1) NPV = V (0) + , (5.6) (1 + DR)t t=1

where V (0) denotes the capitalized impacts in the reference year (2006), and T = 30 is the policy time span. Because baseline and policy scenarios are provided for only a subset of the years in the policy timespan (2006, 2026, 2036), the year-by-year results are obtained by linearly interpolating the monetary impacts between ﬁxed contour years.

We select the quantity of interest for this case study to be the net cost-beneﬁt of each noise stringency option, given as the net present value of the policy-minus-

126 baseline scenario plus the corresponding cost of implementation. This quantity can be written as:

Y = NPVpolicy − NPVbaseline + IC. (5.7)

All monetary values are expressed year 2006 USD. The complexity of Y represents the intrinsic extent of the distribution for the net monetary impacts of the proposed stringency option. The risk associated with Y is deﬁned as P (Y > 0), and can be interpreted as the probability that the stringency option is not cost-beneﬁcial.

5.3 Results and Discussion

5.3.1 Comparison of Stringency Options and Discount Rates

-1 NPV (Billion 2006 USD) (Billion2006 NPV

-2 Str 1 Str 2 Str 3 DR = 3% DR = 5% DR = 7% DR = 9%

Figure 5-3: Comparison of mean estimates for Y in Stringencies 1–3 and discount rates between 3%–9%. Error bars denote 10th, 90th percentile values.

Figure 5-3 shows a comparison of the mean net monetary impacts of Stringencies 1–3 at discount rates of 3%, 5%, 7%, and 9%. Table 5.2 provides the corresponding uncertainty estimates. We see that across all discount rates considered, the mean of Y is negative in Stringency 2, positive in Stringency 3, and mixed in Stringency 1. The complexity of Y remains relatively constant within each stringency, but increases signiﬁcantly between the stringencies. This suggests that an increase in noise stringency is concurrent with a growth in complexity (and thus, a decrease in robustness).

127 Str. Stringency 1 Stringency 2 Stringency 3 DR 3% 5% 7% 9% 3% 5% 7% 9% 3% 5% 7% 9% Mean -0.12 0 0.08 0.13 -1.19 -0.72 -0.41 -0.19 2.91 2.29 1.86 1.56 10th -0.31 -0.19 -0.11 -0.06 -1.42 -0.94 -0.63 -0.42 2.42 1.80 1.37 1.07 90th 0.07 0.19 0.26 0.32 -0.96 -0.50 -0.19 0.03 3.41 2.78 2.35 2.05 SD 0.14 0.14 0.14 0.14 0.17 0.16 0.16 0.16 0.37 0.36 0.36 0.35 Comp. 0.56 0.54 0.53 0.53 0.70 0.66 0.64 0.63 1.49 1.43 1.40 1.38 Risk 24% 49% 66% 77% 0% 0% 0% 15% 100% 100% 100% 100%

Table 5.2: Comparison of mean and uncertainty estimates of Y for Stringencies 1– 3 and discount rates between 3%–9%. All monetary values (mean, 10th and 90th percentiles, standard deviation, and complexity) are listed in 2006 USD (billions).

The risk associated with Y increases with discount rate within Stringency 1, equals zero in Stringency 2 (except when DR= 9%), and remains at 100% in Stringency 3. From these results, we conclude that under the current assumptions, Stringency 2 is virtually always cost-beneﬁcial and Stringency 3 is never cost-beneﬁcial. The most interesting option is Stringency 1, which straddles the border, with risk estimates ranging between 24–77%. Because of this, we will focus our attention on Stringency 1, and in the following section investigate the factors that drive its uncertainty.

5.3.2 Global and Distributional Sensitivity Analysis

Figure 5-4(a) shows global sensitivity analysis results for Stringency 1 when a discount rate of 3% is assumed. With a MSI of 0.93, implementation cost is by far the largest contributor to output variability. On the other hand, the scientific inputs (BNL, CU, and RP), along with factor interactions (Int.), each contributes only a minuscule proportion. This apportionment result does not change when the discount rate is increased (see Table 5.3, left side). Thus, in the remainder of this section, we will focus on the case where DR = 3%. The distributional sensitivity analysis results for this case are shown in Figure 5-5(a). We see that the adjusted main effect sensitivity index for each factor varies linearly with δ. If we permit BNL, CU, RP, and IC to deviate from their nominal distribution families specified in Table 5.1, the resulting effect on adjSi(δ) is negli-

128 2% 3%1% 1% 4%

32% 43% 93% BNL CU 20% BNL RP CU IC RP Int. Int.

(a) IC ∼ U[low est., high est.] (b) IC = mean estimate

Figure 5-4: Main eﬀect sensitivity indices for Stringency 1, DR = 3%, with IC modeled probabilistically in (a) and as a deterministic value in (b)

IC ∼ U[low est., high est.] IC = mean estimate DR 3% 5% 7% 9% 3% 5% 7% 9% BNL 0.02 0.01 0 0 0.32 0.32 0.32 0.32 CU 0.01 0 0 0 0.20 0.20 0.20 0.20 RP 0.03 0.01 0 0 0.43 0.43 0.43 0.43 IC 0.93 0.97 0.99 0.99 N/A N/A N/A N/A Int. 0.01 0.01 0.01 0.01 0.04 0.04 0.04 0.04

Table 5.3: Main eﬀect sensitivity indices for Stringency 1 at various discount rates

gible.10 Thus, we can conclude that irrespective of distribution shape, the relative factor contribution to output variance does not change with δ. Next, we investigate the inﬂuence of our initial assumption that IC is a uniform random variable on the interval delimited by the low and high cost estimates. Figures 5-6(a) through 5-6(c) show the distribution of Y for 10,000 MC samples when IC is assumed to be a uniform, symmetric triangular, or Gaussian random variable on this interval.11 The variance of IC is decreased steadily (by 50% between the uniform and

10In DSA, the regression parameters are modeled as a multivariate Gaussian random variable (MVN) in R2 parameterized by the vector of means M and the covariance matrix Σ. This is an approximation since the RP distribution is not truly Gaussian (as can be seen from Figure 5-1). Furthermore, we have not permitted changes to the RP distribution family, as its shape is speciﬁed according to empirical evidence rather than modeling assumptions. 11Consistent with the approach in Chapter 4, we approximate the Gaussian distribution with

129 1 1 BNL: Tri->Uni BNL: Tri->Tri 0.8 0.8 BNL: Tri->Gaussian BNL: Tri->Bmd Uni

CU: Tri->Uni ) ) 0.6

 0.6

 CU: Tri->Tri

(

( i i CU: Tri->Gaussian

0.4 adjS 0.4 CU: Tri->Bmd Uni adjS IC: Uni->Uni IC: Uni->Tri 0.2 0.2 IC: Uni->Gaussian IC: Uni->Bmd Uni RP: MVN->MVN 0 0 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 1- 1- (a) IC ∼ U[low est., high est.]

0.8 )

δ 0.6 ( i

adjS 0.4

0.2

0 0 0.2 0.4 0.6 0.8 1 1−δ (b) IC = mean estimate

Figure 5-5: Adjusted main eﬀect sensitivity indices for Stringency 1, DR = 3%, with IC modeled probabilistically in (a) and as a deterministic value in (b)

triangular cases, and by 66.7% between the uniform and Gaussian cases), which in

turn reduces the standard deviation and complexity of Y . Since µY < 0, a reduction in var (Y ) also decreases P (Y > 0), hence increasing the probability that the policy will result in a net beneﬁt (i.e., improving reliability).12 Despite these signiﬁ- cant reductions in uncertainty, however, IC still remains the largest driver of output

variability, as its main effect sensitivity index (SIC ) equals 0.86 for the symmetric triangular case and 0.82 for the Gaussian case. In the limiting case, Figure 5-6(d) shows the distribution of Y when IC is fixed at a deterministic value equal to the mean cost estimate for Stringency 1. Compared to the probabilistic cases, there are significant reductions in the standard deviation

a truncated version in which ±3 standard deviations of the distribution fall within the speciﬁed interval. 12We note that this outcome is not necessarily true of other stringencies and discount rates in the CAEP/9 analysis. For example, when DR = 7% in Stringency 1, µY > 0 (see Table 5.2), and thus a decrease in var (Y ) actually increases risk.

130 2500 2500 Mean = $−0.12B Mean = $−0.12B SD = $0.14B SD = $0.10B 2000 2000 Comp = $0.56B Comp = $0.43B Risk = 23.9% Risk = 12.7% 1500 1500 S = 0.93 S = 0.86 IC IC 1000 1000 Number of samples Number of samples 500 500

0 0 −0.6 −0.4 −0.2 0 0.2 0.4 −0.6 −0.4 −0.2 0 0.2 0.4 Net present value (Billion 2006 USD) Net present value (Billion 2006 USD) (a) IC ∼ U[low est., high est.] (b) IC ∼ T [low, high, mean]

2500 2500 Mean = $−0.12B Mean = $−0.11B SD = $0.09B SD = $0.04B 2000 2000 Comp = $0.36B Comp = $0.15B Risk = 7.3% Risk = 0% 1500 1500 S = 0.82 IC 1000 1000 Number of samples Number of samples 500 500

0 0 −0.6 −0.4 −0.2 0 0.2 0.4 −0.6 −0.4 −0.2 0 0.2 0.4 Net present value (Billion 2006 USD) Net present value (Billion 2006 USD) (c) IC ∼ N [mean, (high − low)/6] (d) IC = mean estimate

Figure 5-6: Histogram of Y under various IC assumptions for Stringency 1, DR = 3%

and complexity of Y . Furthermore, the risk of a net policy dis-benefit is reduced to zero. Figure 5-4(b) shows the GSA results for the deterministic IC case. Of the scientific inputs, the regression parameters contribute the largest proportion to output variance at 43%, followed by BNL at 32% and CU at 20%, with interactions again playing a minor role. As in the probabilistic case, the variance apportionment for the deterministic IC case remains constant across all discount rates considered (see Table 5.3, right side). Likewise, the DSA results in Figure 5-5(b) reveal that the adjusted main effect sensitivity indices tend to vary linearly with δ, and incur only small deviations when changes between distribution families are considered.

131 5.3.3 Visualizing Tradeoﬀs

Building off of the variance-based sensitivity analyses from the previous section, we now investigate how changes in the mean or variance of the model inputs affect complexity and risk in the QOI. For this analysis, we focus on BNL, CU, and IC, whose distributions are specified according to engineering assumptions. The regression parameters are excluded, as additional research is needed to extend local sensitivity analysis and visualization techniques to multivariate random variables.

0.2 0.2 0.7 0.8 0.6 0.7 0.5 0.6 0.1 0.1 0.4 0.5 0.3 0.4 0 0 0.2 0.3 0.1 0.2

IC mean ($B) −0.1 IC mean ($B) −0.1 0.1 0.57 0.56

−0.2 0.55 −0.2

50 51 52 53 54 55 50 51 52 53 54 55 BNL mean (dB) BNL mean (dB) (a) Lines of constant complexity due to changes (b) Lines of constant risk due to changes in in mean mean

1 1 0.08 0.08 0.25 0.25 0.8 0.8 0.3 0.3 0.04 0.12 0.04 0.12 0.35 0.35 0.08 0.08 0.6 0.6 0.12 0.4 0.4 0.16 0.12 0.16 0.16 0.16 0.4 0.45 0.45 0.4

0.2 0.2 0.5 0.2 0.2 0.2 0.5 0.2 Percent reduction in var(IC) Percent reduction in var(IC) 0.55 0 0.55 0 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 Percent reduction in var(BNL) Percent reduction in var(BNL) (c) Lines of constant complexity due to (d) Lines of constant risk due to reductions in reductions in variance variance

Figure 5-7: Uncertainty contours associated with changes in the mean and variance of BNL and IC, generated from 10,000 MC simulations (solid colored lines) or approximated using sensitivity analysis results (dashed black lines)

In Figures 5-7 through 5-9, we let the mean of BNL, CU, and IC vary between the

132 0.8 0.7 0.2 0.2 0.6 0.7 0.6 0.5 0.1 0.1 0.5 0.4 0.4 0.3 0 0 0.3 0.2

0.2 0.1

IC mean ($B) −0.1 IC mean ($B) −0.1 0.1 0.55 0.57

−0.2 0.56 −0.2

−2 −1 0 1 2 −2 −1 0 1 2 CU mean (dB) CU mean (dB) (a) Lines of constant complexity due to changes (b) Lines of constant risk due to changes in in mean mean

1 1 0.08 0.08 0.25 0.25 0.8 0.3 0.3 0.8 0.04 0.12 0.04 0.12 0.35 0.35 0.08 0.08 0.6 0.6 0.4 0.4 0.160.12 0.12 0.16

0.16 0.16 0.4 0.45 0.45 0.4

0.2 0.2 0.5 0.2 0.2 0.2 0.5 0.2 Percent reduction in var(IC) Percent reduction in var(IC)

0.55 0.55 0 0 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 Percent reduction in var(CU) Percent reduction in var(CU) (c) Lines of constant complexity due to (d) Lines of constant risk due to reductions in reductions in variance variance

Figure 5-8: Uncertainty contours associated with changes in the mean and variance of CU and IC lower and upper bounds of their respective distributions, as well as allow their variance to be reduced by 0–100% (assuming that such reductions decrease the extent of the underlying distribution symmetrically about the mean). No changes in distribution family are considered. The solid colored lines denote contours generated using 10,000 MC simulations at each design parameter combination, whereas the dashed black lines indicate approximations based on distributional, local, or entropy power-based sensitivity analysis results. A black X in each figure designates the location of the initial case as defined in Table 5.1. Figures 5-7 and 5-8 show how changes in IC trade off against BNL and CU, respectively. Figures 5-7(a) and 5-8(a) reveal that varying the mean implementation

133 2 2

1.5 0.59 0.56 1.5 0.15 0.25

1 1 0.1 0.3 0.57 0.55 0.2 0.5 0.58 0.5

0 0

−0.5 0.56 −0.5 0.15 0.25

CU mean (dB) 0.54 CU mean (dB) 0.35 −1 −1 0.3 0.2 −1.5 0.55 −1.5

0.57 −2 −2 50 51 52 53 54 55 50 51 52 53 54 55 BNL mean (dB) BNL mean (dB) (a) Lines of constant complexity due to changes (b) Lines of constant risk due to changes in in mean mean

1 1

0.237 0.238 0.55 0.55 0.8 0.8 0.238 0.237

0.6 0.6 0.237 0.237

0.238 0.236 0.4 0.4 0.237 0.237 0.56 0.55

0.2 0.2 0.237 Percent reduction in var(CU) Percent reduction in var(CU)

0 0 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 Percent reduction in var(BNL) Percent reduction in var(BNL) (c) Lines of constant complexity due to (d) Lines of constant risk due to reductions in reductions in variance variance

Figure 5-9: Uncertainty contours associated with changes in the mean and variance of BNL and CU cost estimate has no effect on output complexity. This is unsurprising since IC is simply added to the NPV of the policy-minus-baseline noise impacts according to (5.7), and thus contributes only a constant shift to the estimates of Y . However, when the mean value of IC is decreased, the risk of incurring a net policy dis-benefit is greatly reduced (Figures 5-7(b) and 5-8(b)). A reduction in the mean BNL value slightly increases complexity and decreases risk, whereas a reduction in the mean CU value slightly decreases complexity and increases risk. This incongruity can be explained by (5.4), which states that a decrease in BNL results in a larger value of ∆dB and thus a higher level of estimated noise impacts, whereas a decrease in CU has the opposite effect. Figures 5-7(c), 5-7(d), 5-8(c), and 5-8(d) show that reducing

134 the variance of IC greatly decreases both complexity and risk in the QOI. This effect is so dominant that reductions in the variance of BNL or CU have negligible impact by comparison. These results corroborate the findings of variance-based sensitivity analysis, and confirm that the most effectual strategy for reducing uncertainty in policy net cost-benefit is to invest additional resources toward improving estimates of implementation cost. Finally, Figure 5-9 illustrates the tradeoffs in output complexity and risk associated with changes in the mean or variance of BNL and CU. Consistent with the trends in Figures 5-7(a), 5-7(b), 5-8(a), and 5-8(b), we see that complexity decreases when the mean of BNL is increased, or when the mean of CU is decreased, whereas risk decreases when the mean of BNL is decreased, or when the mean of CU is increased. Figures 5-9(c) and 5-9(d) show that reductions in the variance of BNL and/or CU have little effect on complexity, and virtually no effect on risk. These results validate our conclusion that from an uncertainty mitigation point of view, it is not worthwhile to expend resources to learn more about background noise level or noise contour uncertainty; instead, efforts should be directed toward reducing uncertainty in policy implementation cost.

5.4 Chapter Summary

In this chapter, we applied the uncertainty and sensitivity analysis methods developed in this thesis to study the net cost-benefit of the CAEP/9 noise stringency options as estimated using the APMT-Impacts Noise Module. Based on our analysis, we conclude that among all sources of uncertainty considered, scenario uncertainty in the form of forecasted policy implementation cost outpaces scientific uncertainty in the noise monetization model by an order of magnitude. This result holds true across all stringencies and discount rates considered, as well as for several reasonable spec- ifications of input distribution shape. Additional sensitivity analyses revealed that as defined, the scientific inputs of the APMT-Impacts Noise Module comprise only a small contribution to the complexity and risk associated with the net cost-benefit of

135 the proposed policy options. For Stringency 1, we recommend that the best course of action for ensuring net policy benefit is to improve economic forecasting capabilities and obtain more precise estimates of implementation cost. In this case study, we have also highlighted the disparity in uncertainty estimates for policy cost-benefit when implementation cost is modeled probabilistically versus as a deterministic value, which represents an extension to previous cost-benefit analyses of the CAEP/9 noise stringency options. We note, however, that a key difference between this work and the prior analysis by Wolfe and Waitz lies in having modeled only the monetized noise impacts of the proposed stringency options, such that the net present value estimates do not capture the full suite of environmental benefits associated with policy implementation. A natural extension of the current work is to incorporate the climate and air quality co-benefits of the CAEP/9 noise stringency options, as well as characterize the effects of uncertainty in those estimates.

136 Chapter 6

Conclusions and Future Work

The overarching goal of this thesis was to develop methods that help quantify, understand, and mitigate the eﬀects of uncertainty in the design of engineering systems. In this chapter, we summarize the progress made toward the thesis objectives, highlight key intellectual contributions, and suggest directions for future research.

6.1 Thesis Summary

For many engineering systems, current design methodologies do not adequately quantify and manage uncertainty as it arises during the design process, which can lead to unacceptable risks, increases in programmatic cost, and schedule overruns. In this thesis, we developed new methods for uncertainty quantification and sensitivity analysis, which can be used to better understand and mitigate the effects of uncertainty. In Chapter 2, we defined complexity as the potential of a system to exhibit unexpected behavior in the quantities of interest, and proposed the use of exponential entropy to quantify complexity in the design context. This complexity metric can be used alongside other measures of uncertainty, such as variance and risk, to afford a richer description of system robustness and reliability. In Chapter 3, we extended existing variance-based global sensitivity analysis to derive an analogous interpretation of uncertainty apportionment based on entropy power, which is proportional to complexity squared. This derivation also revealed

137 that a factor’s contribution to output entropy power consists of two effects, which are related to its spread (as characterized by intrinsic extent) versus distribution shape (as characterized by non-Gaussianity). To further examine the influence of input distribution shape on estimates of output variance, we also broadened the existing distributional sensitivity analysis methodology to accommodate changes to distribution family. Another key aspect the work in this thesis is the development of local sensitivity analysis techniques to predict changes in the complexity and risk of the quantities of interest resulting from various design modifications. These local approximations are particularly useful for systems whose simulation models are computationally expensive, as they can be used to identify tradeoffs and infer trends in the design space without performing additional model evaluations. Furthermore, they can be connected with budgets for uncertainty and cost in order to identify options for improving robustness and reliability and inform decisions regarding resource allocation. In Chapters 4 and 5, we demonstrated the uncertainty and sensitivity analysis methods developed in this thesis on two engineering examples.

6.2 Thesis Contributions

The main contributions of this thesis are:

• The proposal of an information entropy-based complexity metric, which can be used in the design context to supplement existing measures of uncertainty with respect to speciﬁed quantities of interest;

• The development of local sensitivity analysis techniques that can approximate changes in complexity and risk resulting from modiﬁcations to design parameter mean or variance, as well as enable the visualization of design tradeoﬀs;

• The identiﬁcation of intrinsic extent and non-Gaussianity as distinct features of a probability distribution that contribute to complexity, as well as the development of techniques to analyze the eﬀect of distribution shape on uncertainty

138 estimates;

• A demonstration of how uncertainty estimates and sensitivity information can be combined with design budgets to guide decisions regarding the allocation of resources to improve system robustness and reliability.

6.3 Future Work

There are a number of possible extensions to the research presented in this thesis. The first is to broaden the application of the uncertainty and sensitivity analysis methods to problems of higher dimensionality. Although the theory underlying many of the techniques (e.g., estimation of complexity and entropy power, characterizing non-Gaussianity using K-L divergence) can easily be extended to higher dimensions, practical issues often hinder their applicability. For example, density estimation — upon which entropy estimation is prefaced — is typically straightforward in 1-D; for higher dimensions, however, it becomes much more challenging, or even downright intractable. High-dimensional density estimation is an active area of research; it is expected that advancements in this field will allow the methods developed in this thesis to be applied to larger classes of engineering problems. A related issue in computational tractability is the expense of pseudo-random Monte Carlo simulation. One avenue of future work could lead to the incorporation of efficient sampling methods, such as Latin hypercube or quasi-random Monte Carlo, to reduce computational cost. In particular, it would be interesting to study how such techniques can be combined with acceptance/rejection sampling or importance sampling to enable sample reuse in distributional sensitivity analysis. In this thesis, we focused primarily on characterizing the downstream effects of parameter uncertainty, which is due to not knowing the true values of the model inputs. Another important area for future research is to extend the proposed system design framework to consider other sources of uncertainty, such as model inadequacy and observation error. This issue is particularly relevant for systems such as the CAEP/9 noise stringency case study presented in Chapter 5, in which we discovered that by

139 modeling implementation cost probabilistically, we identified a major contributor to uncertainty in policy net cost-benefit. Within the context of the APMT-Impacts Noise Module, other potential sources of uncertainty include the spatial and temporal resolution of the AEDT noise contours (which are currently provided in increments of 5 dB for only three of the years in the 30-year policy duration), as well as changes in population and income level during the analysis timespan. By constructing a more accurate picture of the various sources of uncertainty present in a system, we can better understand their effects on the quantities of interest, as well as take steps to mitigate potentially detrimental impacts in order to ensure system robustness and reliability.

140 Appendix A

Bimodal Distribution with Uniform Peaks

In Section 1.2, we introduced a bimodal distribution consisting of two uniform peaks

(i.e., a “bimodal uniform” distribution), parameterized by six quantities: aL, bL, fL, aR, bR, and fR. Below, we reproduce Figure 1-3(d) as well as the expressions for the

mean and variance of such a distribution from Table 2.1.

푥 푋 푓 푓퐿 푓푅

푎퐿 푏퐿 푎푅 푏푅 푥

Figure A-1: Bimodal uniform distribution with peak bounds aL, bL, aR, and bR, and peak heights fL and fR

f f µ = L (b2 − a2 ) + R (b2 − a2 ) (A.1) 2 L L 2 R R f f σ2 = L (b3 − a3 ) + R (b3 − a3 ) − ( [X])2 (A.2) 3 L L 3 R R E

141 A.1 Specifying Distribution Parameters

Because there are five degrees of freedom in the parameter specification, mean and variance alone are not sufficient to uniquely define a bimodal uniform random variable. Thus, we must make additional assumptions about the shape of the distribution. In this thesis, we assume that the two peaks are symmetric about the mean and that each peak has width equal to 1/4 of the extent of a uniform random variable with

the same variance. This allows us to compute aL, bL, fL, aR, bR, and fR using the following procedure.

Algorithm 1 Specify a bimodal uniform random variable X given µ and σ 1. Find the lower and upper bound of a uniform distribution with mean µ and standard deviation σ: √ b − a = σ 12 ⇒ a = µ − (b − a)/2, b = µ + (b − a)/2

2. Assume that the peaks of X are symmetric, each with a width equal to (b−a)/4. Thus, the peak heights are fL = fR = 2/(b − a).

∗ ∗ 3 3. Let bR = b and aR = 4 (b − a) + a.

∗ ∗ ∗ 4. Find aL and bL such that the bimodal uniform random variable X ∼ ∗ ∗ ∗ ∗ 2 BU{aL, bL, fL, aR, bR, fR} has a variance of σ . This requires solving the following system of equations:

∗ ∗ (b − a)/4 = bL − aL f f f f 2 σ2 = L (b∗3 − a∗3) + R (b∗3 − a∗3) − L (b∗2 − a∗2) + R (b∗2 − a∗2) 3 L L 3 R R 2 L L 2 R R

∗ ∗ ∗ ∗ ∗ 5. The mean of X is µ = (bL + aR)/2. Compute the diﬀerence ∆µ = µ − µ. ∗ ∗ ∗ ∗ Then, shift each of aL, bL, aR, and bR by ∆µ to obtain aL, bL, aR, and bR:

∗ ∗ aL = aL − ∆µ aR = aR − ∆µ ∗ ∗ bL = bL − ∆µ bR = bR − ∆µ

6. Deﬁne X ∼ BU(aL, bL, fL, aR, bR, fR).

142 A.2 Options for Variance Reduction

Algorithm 1 can be used to deﬁne a symmetric bimodal uniform random variable BU o o o o o o o o o2 X ∼ BU(aL, bL, fL, aR, bR, fR) with mean µ and variance σ . For a given value of δ, we desire to specify parameters for a new bimodal uniform random variable X0 ∼ 0 0 0 0 0 0 02 o2 BU(aL, bL, fL, aR, bR, fR) such that σ = δσ . Some possible options for variance reduction include:

1. Decreasing the width of one or both peaks.

2. Decreasing the separation between the peaks.

3. Increasing the height of one peak versus the other.

In this section, we will focus on Options 2 and 3, as Option 1 is typically not suﬃcient to reduce variance to the requisite level for most values of δ. Algorithms 2 and 3 provide the procedures to deﬁne X0 using Options 2 and 3, respectively. Note that neither option is valid for the full range of possible values δ ∈ [0, 1]; instead, each option is applicable for δ ∈ (δ∗, 1]. Using Option 2, the minimum achievable variance occurs when there is no separation between the peaks (i.e., the symmetric bimodal uniform distribution becomes 0 0 02 o o 2 a uniform distribution), such that bL = aR. In this case, σ = (bL − aL) /3 = o o 2 (bR − aR) /3. Similarly, using Option 3, variance is minimized when the height of one peak is reduced to zero while that of the other is increased, such that σ02 = o o 2 o o 2 ∗ (bL −aL) /12 = (bR −aR) /12. The values of δ for these options are listed below. Fi- nally, we note that Algorithm 1 can also be used to specify a new symmetric bimodal uniform random variable X0 based on XBU o for all values of δ ∈ [0, 1].

Algorithm 1: δ∗ = 0 (bo − ao )2 Algorithm 2: δ∗ = L L 3σo2 (bo − ao )2 Algorithm 3: δ∗ = L L 12σo2

143 Algorithm 2 Decrease peak separation to achieve σ02 = δσo2 0 o 0 o 1. Set peak heights fL = fL and fR = fR. 2. Select at random which peak to move. If the left peak is chosen, go to Step 3. Else, go to Step 4.

0 o 0 o 0 0 3. Set aR = aR and bR = bR. Find aL and bL by solving the following system of equations:

o 0 0 fR o o bL − aL = o (bR − aR) fL f o f o f o f o 2 δσo2 = L (b03 − a03) + R (bo3 − ao3) − L (b02 − a02) + R (bo2 − ao2) 3 L L 3 R R 2 L L 2 R R

0 o 0 o 0 0 4. Set aL = aL and bL = bL. Find aR and bR by solving the following system of equations:

o 0 0 fL o o bR − aR = o (bL − aL) fR f o f o f o f o 2 δσo2 = L (bo3 − ao3) + R (b03 − a03) − L (bo2 − ao2) + R (b02 − a02) 3 L L 3 R R 2 L L 2 R R

0 0 0 0 0 0 0 5. Deﬁne X ∼ BU(aL, bL, fL, aR, bR, fR).

Algorithm 3 Vary peak heights achieve σ02 = δσo2 0 o 0 o 0 o 0 o 1. Set peak bounds aL = aL, bL = bL, aR = aR, and bR = bR,

0 0 2. Find fL and fR by solving the following system of equations:

0 o o 0 o o 1 = fL(bL − aL) + fR(bR − aR) f 0 f 0 f 0 f 0 2 δσo2 = L (bo3 − ao3) + R (bo3 − ao3) − L (bo2 − ao2) + R (bo2 − ao2) 3 L L 3 R R 2 L L 2 R R

3. Select at random which peak height to decrease. If the left peak is chosen, set 0 0 0 0 fL equal to the smaller of {fL, fR} obtained in Step 2, and fR equal to the 0 0 0 0 larger. Otherwise, set fR to equal the smaller of {fL, fR}, and fL equal to the larger.

0 0 0 0 0 0 0 4. Deﬁne X ∼ BU(aL, bL, fL, aR, bR, fR).

144 Figure A-2 shows a bimodal uniform distribution XBU o obtained using Algorithm 1 for µo = 0.5 and σo = 0.29 as well as several examples of bimodal uniform distributions X0 obtained using Algorithms 1–3 for δ = 0.5. Each realization of X0 has σ0 = 0.20.

5 o Alg. 1 (XBU ) Alg. 1 (X') 4 Alg. 2 (X') Alg. 3 (X')

2 Probability density

0 0 0.2 0.4 0.6 0.8 1 x

Figure A-2: Examples of distributions for X0 obtained using Algorithms 1–3 (solid lines), based on an initial distribution XBU o (dashed green line) and δ = 0.5

A.3 Switching to a Bimodal Uniform Distribution

In this section, we outline the procedure for switching from a uniform, triangular, or Gaussian random variable Xo to a bimodal uniform random variable X0. As in Appendix 3.2, the key steps are to estimate the parameters of X0 and to specify κ so that the acceptance/rejection method can be used to generate samples of X0 from samples of Xo. Examples of reasonable bimodal uniform distributions generated using the procedure below are shown in Figure A-3.

0 0 0 0 0 0 Selecting aL, bL, fL, aR, bR, and fR: BU o o o o o o o o 1. Use Algorithm 1 to deﬁne X ∼ BU(aL, bL, fL, aR, bR, fR) based on µ , σo.

2. Determine which of Algorithms 1–3 can be used to achieve σ02 = δσo2.

145 3. Of the possible algorithms, select one at random and use it to specify X0 ∼ 0 0 0 0 0 0 BU(aL, bL, fL, aR, bR, fR).

Specifying κ: 0 0 0 0 • Evaluate fXo (x) at each of aL, bL, aR, and bR. n 0 0 0 0 o fL fL fR fR • Set κ = max 0 , 0 , 0 , 0 . fXo (aL) fXo (bL) fXo (aR) fXo (bR)

3.5 7

3 6

2.5 5

2 4

1.5 3

Probability density 1 Probability density 2

0.5 1

0 0 −0.2 0 0.2 0.4 0.6 0.8 1 1.2 0 0.2 0.4 0.6 0.8 1 x x (a) Uniform to bimodal uniform: Xo ∼ U[0, 1], (b) Gaussian to uniform: Xo ∼ T (0, 1, 0.6), δ = 0.7, k = 1 δ = 0.3, k = 1

2.5

1.5

1 Probability density

0.5

0 −3 −2 −1 0 1 2 3 x (c) Gaussian to triangular: Xo ∼ N (0, 1), δ = 0.9, k = 1

Figure A-3: Examples of switching from a uniform, triangular, or Gaussian distribution to a bimodal uniform distribution. In each ﬁgure, the solid red line denotes fXo (x), the dashed blue line represents fX0 (x), and the dotted black line shows κfXo (x), which is greater than or equal to fX0 (x) for all x.

146 Appendix B

Derivation of Local Sensitivity Analysis Results

In Section 3.3, we introduced expressions for the sensitivity of complexity and risk to local changes in the mean or standard deviation of the quantity of interest. Here, we provide the derivations for (3.15), (3.16), (3.19), and (3.20). Letting Y and Y 0 represent initial and new estimates of the QOI, respectively, the corresponding changes in mean, standard deviation, complexity, and risk, are given below:

∆µ = µY 0 − µY ,

∆σ = σY 0 − σY , ∆C = C(Y 0) − C(Y ),

∆P = P (Y 0 < r) − P (Y < r).

B.1 Perturbations in Mean

First, we consider the case where Y and Y 0 are related by a constant shift ∆µ, while ∆σ = 0: Y 0 = Y + ∆µ. (B.1)

147 B.1.1 Eﬀect on Complexity

A constant shift has no eﬀect on a random variable’s diﬀerential entropy, as h(Y + ∆µ) = h(Y ) [88]. Therefore, the exponential entropy of Y and Y 0 are also equivalent:

C(Y 0) = exp[h(Y + ∆µ)] = C(Y ). (B.2)

Since ∆C = 0 for all values of ∆µ, we have that the partial derivative of C(Y ) to µY is zero: ∂C(Y ) = 0. (B.3) ∂µY

B.1.2 Eﬀect on Risk

Here, we deﬁne a new variable z = y − ∆µ, and note that the probability densities of Y and Y 0 are related by:

fY 0 (y) = fY (y − ∆µ) = fY (z). (B.4)

Therefore, the failure probability associated with Y 0 is:

Z r Z r 0 P (Y < r) = fY 0 (y) dy = fY (z) dy. (B.5) −∞ −∞

0 Taking the partial derivative of P (Y < r) with respect to µY 0 , and noting that

∂z/∂µY 0 = ∂[y − (µY 0 − µY )]/∂µY 0 = −1, we obtain:

∂P (Y 0 < r) Z r ∂f (z) ∂z Z r ∂f (z) = Y dy = − Y dy. (B.6) ∂µY 0 −∞ ∂z ∂µY 0 −∞ ∂z

In the limit as ∆µ → 0, z → y and µY 0 → µY . Then, the above expression becomes:

∂P (Y < r) Z r ∂f (y) r Y = − dy = −fY (y) . (B.7) ∂µY −∞ ∂y −∞

148 Commonly, fY (y) → 0 as y → −∞, so we have:

∂P (Y < r) = −fY (r). (B.8) ∂µY

B.2 Perturbations in Standard Deviation

Next, we shrink the standard deviation of Y while leaving the mean unchanged. For this derivation, we scale Y by a positive constant γ, and deﬁne new random variable Y 0 such that:1

Y − µ Y 0 = Y , (B.9) γ

fY 0 (y) = γfY (γy + µY ). (B.10)

0 Note that we have included in Y a constant shift of ∆µ = −µY , such that µY 0 = 0. This shift serves to recenter Y 0 at the origin so as to simplify the derivation; in general we do not restrict Y 0 to be zero mean. To account for the shift, we adjust 0 0 the requirement on Y accordingly by deﬁning a new constant r = r − µY . We also 0 deﬁne a new variable z = γy + µY . The variance of Y is then given by:

Z ∞ Z ∞ 2 2 2 1 2 σY σY 0 = γ y fY (γy + µY ) dy = 2 (z − µY ) fY (z) dz = 2 , (B.11) −∞ γ −∞ γ which corresponds to the following change in standard deviation:

σ 1 ∆σ = Y − σ = σ − 1 . (B.12) γ Y Y γ

Thus, for γ > 1, standard deviation decreases between Y and Y 0. The diﬀerential entropy estimates h(Y ) and h(Y 0) are related by:

Z ∞ 0 h(Y ) = − fY (z) log[kfY (z)] dz = h(Y ) − log γ. (B.13) −∞

1Note that in Sections 2.3 and 3.4.2, we described a multiplicative scaling of Y as Y 0 = αY or 0 Y = α(Y − µY ); in this section we use γ for convenience, where γ = 1/α.

149 B.2.1 Eﬀect on Complexity

The change in complexity is given by:

exp[h(Y )] 1 ∆C = − exp{h(Y )} = C(Y ) − 1 . (B.14) exp[log γ] γ

Combining (B.12) and (B.14), we obtain:

∆C C(Y ) = . (B.15) ∆σ σY

In the limit as ∆σ → 0, we have:

∂C(Y ) C(Y ) = . (B.16) ∂σY σY

B.2.2 Eﬀect on Risk

The probability of failure associated with Y 0 is:

Z r0 0 0 0 P (Y < r ) = γ fY (γy + µY ) dy = P (Y < γr + µY ). (B.17) −∞

Taking the partial derivative with respect to γ, we obtain:

0 0 Z γr +µY ∂P (Y < γr + µY ) ∂ 0 0 = fY (z) dz = r fY (γr + µY ). (B.18) ∂γ ∂γ −∞

0 Then, if we let γ → 1 and make the substitution r = r − µY , we have:

∂P (Y < r) = (r − µ )f (r). (B.19) ∂γ Y Y

We seek to compute the partial derivative of P (Y < r) with respect to σY . Using the chain rule, we obtain:

∂P (Y < r) ∂P (Y < r) ∂γ = (B.20) ∂σY ∂γ ∂σY

150 Recall that from (B.11), γσY 0 = σY . Diﬀerentiating both sides with respect to σY 0 allows us to estimate ∂γ/∂σY :

∂γ γ + σY 0 = 0, (B.21) ∂σY 0 ∂γ −γ = . (B.22) ∂σY 0 σY 0

As γ → 1, σY 0 → σY , and the above expression becomes:

∂γ −1 = . (B.23) ∂σY σY

Combining (B.19), (B.20), and (B.23), we have:

∂P (Y < r) (µY − r) = fY (r). (B.24) ∂σY σY

Finally, we note that in the case where risk is expressed as P (Y > r) = 1−P (Y < r), the corresponding sensitivities ∂P (Y > r)/∂µY and ∂P (Y > r)/∂σY are related to (B.8) and (B.24), respectively, through only a change in sign.

151

Appendix C

Derivation of the Entropy Power Decomposition

In Section 3.4.1, we introduced the following entropy power decomposition, which apportions N(Y ), the entropy power of the QOI, into contributions due to the entropy power and non-Gaussianity of the auxiliary random variables Z1,...,Zm and their interactions:

G X G N(Y )exp[2DKL(Y ||Y )] = N(Zi) exp[2DKL(Zi||Zi )] i X G + N(Zij) exp[2DKL(Zij||Zij )] + ... i

G + N(Z12...m) exp[2DKL(Z12...m||Z12...m)]. (C.1)

The superscript G in the above expression denotes equivalent Gaussian random variables that have the same mean and variance as the original distribution.

In this section, we derive the entropy power decomposition starting from the ANOVA-HDMR given in (3.1) and (3.40). We limit our consideration to a system consisting of only two input factors (m = 2), although the results can be generalized

153 to higher dimensions. For such a system, (3.40) and (3.41) reduce to:

Y = g0 + Z1 + Z2 + Z12, (C.2)

var (Y ) = var (Z1) + var (Z2) + var (Z12). (C.3)

The auxiliary random variables Z1, Z2, and Z12 are uncorrelated, though not necessarily independent.1 For this two-factor system, the distributions of the equivalent Gaussian random variables and the corresponding probability densities are:

2 G 1 −(z − µZ1 ) Z1 ∼ N (µZ1 , σZ1 ) fZG (z) = exp 2 1 q 2 2σ 2πσ Z1 Z1 2 G 1 −(z − µZ2 ) Z2 ∼ N (µZ2 , σZ2 ) fZG (z) = exp 2 2 q 2 2σ 2πσ Z2 Z2 2 G 1 −(z − µZ12 ) Z12 ∼ N (µZ12 , σZ12 ) fZG (z) = exp 2 12 q 2 2σ 2πσ Z12 Z12 2 G 1 −(z − µY ) Y ∼ N (µ , σ ) f G (z) = exp Y Y Y p 2 2σ2 2πσY Y

To verify the entropy power decomposition, we need to show that:

G G G G exp[2h(Y,Y )] = exp[2h(Z1,Z1 )] + exp[2h(Z2,Z2 )] + exp[2h(Z12,Z12)]. (C.4)

G where h(Z1,Z1 ) represents the cross entropy between Z1 and its equivalent Gaussian G G random variable Z1 (likewise Y , Z2, and Z12). By deﬁnition, h(Z1,Z1 ) is computed as:

1 Since Z12 is a function of both X1 and X2, it is necessarily dependent on both Z1 and Z2, which are respectively functions of X1 and X2 alone. However, Z1 and Z2 are independent, due to our assumption in Section 1.2 that all design parameters are independent.

154 Z ∞ G h(Z1,Z ) = − fZ (z) log f G (z) dz (C.5) 1 1 Z1 −∞   Z ∞ 2  1 −(z − µZ1 )  = − fZ1 (z) log exp 2 dz (C.6) q 2 2σ −∞  2πσ Z1  Z1   Z ∞ 1 = − f (z) log dz Z1 q  −∞ 2πσ2 Z1 Z ∞ −(z − µ )2 − f (z) log exp Z1 dz (C.7) Z1 2σ2 −∞ Z1 q Z ∞ Z ∞ 2 1 2 = log 2πσ fZ (z) dz + (z − µZ ) fZ (z) dz. (C.8) Z1 1 2σ2 1 1 −∞ Z1 −∞

Z ∞ Z ∞ 2 2 Noting that fZ1 (z) dz = 1 and (z − µZ1 ) fZ1 (z) dz = σZ1 , (C.8) simpliﬁes to: −∞ −∞

q G 2 1 h(Z1,Z ) = log 2πσ + . (C.9) 1 Z1 2

Multiplying both sides by two and taking the exponential, we get:

q G 2 1 exp[2h(Z1,Z )] = exp 2 log 2πσ + 2 (C.10) 1 Z1 2 2 = exp log(2πσZ1 ) + 1 (C.11) 2 = 2πeσZ1 . (C.12)

Similarly, the other terms of (C.4) are given by:

G 2 exp[2h(Z2,Z2 )] = 2πeσZ2 , (C.13) G 2 exp[2h(Z12,Z12)] = 2πeσZ12 , (C.14) G 2 exp[2h(Y,Y )] = 2πeσY . (C.15)

155 Substituting (C.12)–(C.15) into (C.4), we obtain:

2 2 2 2 2πeσY = 2πeσZ1 + 2πeσZ2 + 2πeσZ12 , (C.16) 2 2 2 2 σY = σZ1 + σZ2 + σZ12 . (C.17)

We have already established that (C.17) is true, as it is equivalent to (C.3). We conclude therefore that (C.4) also holds true. Using (2.24) to relate cross entropy to diﬀerential entropy and K-L divergence, we can rewrite (C.4) as follows:

G G exp[2h(Y ) + 2DKL(Y ||Y )] = exp[2h(Z1) + 2DKL(Z1||Z1 )]

G + exp[2h(Z2) + 2DKL(Z2||Z2 )]

G + exp[2h(Z12) + 2DKL(Z12||Z12)]. (C.18)

Using (2.26), we can rewrite the above expression in terms of N(Y ):

G G (2πe)N(Y ) exp[2DKL(Y ||Y )] = (2πe)N(Z1) exp[2DKL(Z1||Z1 )]

G + (2πe)N(Z2) exp[2DKL(Z2||Z2 )]

G + (2πe)N(Z12) exp[2DKL(Z12||Z12)]. (C.19)

Dividing both sides by (2πe), the result becomes:

G G N(Y ) exp[2DKL(Y ||Y )] = N(Z1) exp[2DKL(Z1||Z1 )]

G + N(Z2) exp[2DKL(Z2||Z2 )] G exp[2DKL(Z12||Z12)] + N(Z12) G , (C.20) exp[2DKL(Y ||Y )] which is exactly (C.1) for m = 2. Thus, we have veriﬁed the entropy power decomposition from Section 3.4.1.

156 Appendix D

CAEP/9 Analysis Airports

Table D.1 lists the 99 US airports used in the CAEP/9 noise stringency case study in Chapter 5, along with the average personal income in the corresponding Metropolitan Statistical Area obtained from the US Bureau of Economic Analysis [156]. Each airport is identiﬁed using its four-letter ICAO designation. All income ﬁgures are in 2006 USD.

No. Airport City State Income 1 KABE Allentown PA 35,717 2 KABQ Albuquerque NM 32,935 3 KACK Nantucket MA 71,837 4 KALB Albany NY 37,828 5 KATL Atlanta GA 39,160 6 KAUS Austin TX 37,022 7 KBDL Hartford CT 46,456 8 KBFI Seattle (Boeing Field) WA 47,006 9 KBHM Birmingham AL 38,087 10 KBNA Nashville TN 38,471 11 KBOI Boise ID 35,591 12 KBOS Boston MA 52,018 13 KBUF Buﬀalo NY 33,402 14 KBWI Baltimore MD 44,704 15 KCAE Columbia SC 33,330 16 KCLE Cleveland OH 38,156 17 KCLT Charlotte NC 39,542 18 KCMH Columbus OH 36,695

157 No. Airport City State Income 19 KCOS Colorado Springs CO 35,649 20 KCVG Cincinnati OH 38,096 21 KDAL Dallas (Love Field) TX 40,614 22 KDAY Dayton OH 33,649 23 KDCA Washington (Reagan) DC 53,384 24 KDFW Dallas/Fort Worth TX 40,614 25 KDSM Des Moines IA 40,432 26 KDTW Detroit MI 37,928 27 KELP El Paso TX 24,718 28 KEWR Newark NJ 50,843 29 KFAT Fresno CA 29,111 30 KFLL Fort Lauderdale FL 42,148 31 KGEG Spokane WA 31,626 32 KGON Groton/New London CT 41,792 33 KGRR Grand Rapids MI 32,734 34 KGSO Greensboro NC 33,978 35 KHOU Houston (Hobby) TX 42,984 36 KHUF Terre Haute IN 26,936 37 KIAD Washington (Dulles) DC 53,384 38 KIAH Houston (Bush) TX 42,984 39 KILN Wilmington OH 31,423 40 KIND Indianapolis IN 38,338 41 KJAX Jacksonville FL 39,197 42 KJFK New York (Kennedy) NY 50,843 43 KLAS Las Vegas NV 38,183 44 KLAX Los Angeles CA 42,329 45 KLGA New York (LaGuardia) NY 50,843 46 KLIT Little Rock AR 35,840 47 KMCI Kansas City MO 38,914 48 KMCO Orlando (International) FL 34,625 49 KMDW Chicago (Midway) IL 43,294 50 KMEM Memphis TN 36,417 51 KMHT Manchester NH 43,160 52 KMIA Miami FL 42,148 53 KMKE Milwaukee WI 41,162 54 KMMH Mammoth Lakes CA 39,404 55 KMSP Minneapolis/St. Paul MN 45,002 56 KMSY New Orleans LA 42,416

158 No. Airport City State Income 57 KOAK Oakland CA 59,640 58 KOKC Oklahoma City OK 36,357 59 KOMA Omaha NE 40,476 60 KONT Ontario CA 29,330 61 KORD Chicago (O’Hare) IL 43,294 62 KORF Norfolk VA 36,567 63 KORL Orlando (Executive) FL 34,625 64 KPDX Portland OR 38,416 65 KPHL Philadelphia PA 43,548 66 KPHX Phoenix AZ 37,173 67 KPIT Pittsburgh PA 38,808 68 KPVD Providence RI 37,777 69 KRDU Raleigh/Durham NC 39,215 70 KRIC Richmond VA 39,730 71 KRNO Reno NV 43,803 72 KROC Rochester NY 35,385 73 KRSW Fort Myers FL 41,089 74 KSAN San Diego CA 43,967 75 KSAT San Antonio TX 32,553 76 KSDF Louisville KY 35,954 77 KSEA Seattle/Tacoma WA 47,006 78 KSFO San Francisco CA 59,640 79 KSHV Shreveport LA 33,418 80 KSJC San Jose CA 56,124 81 KSLC Salt Lake City UT 36,906 82 KSMF Sacramento CA 38,852 83 KSRQ Sarasota FL 47,497 84 KSTL St. Louis MO 39,126 85 KSWF Newburgh NY 36,563 86 KSYR Syracuse NY 32,574 87 KTEB Teterboro NJ 50,843 88 KTOL Toledo OH 32,812 89 KTPA Tampa FL 36,470 90 KTUL Tulsa OK 38,470 91 KTUS Tucscon AZ 33,263 92 KTYS Knoxville TN 33,150 93 KVNY Van Nuys CA 42,329 94 PANC Anchorage AK 42,256

159 No. Airport City State Income 95 PHKO Kailua-Kona HI 29,185 96 PHLI Lihue HI 32,442 97 PHNL Honolulu HI 39,938 98 PHOG Kahului HI 34,160 99 PHTO Hilo HI 29,185

Table D.1: CAEP/9 US airports and income levels

160 Bibliography

[1] P. Eremenko. Formal Model-Based Design & Manufacture: A Template for Managing Complexity in Large-Scale Cyber-Physical Systems. In Conference on Systems Engineering Research, Atlanta, GA, 2013.

[2] L. Allen, R. Angel, J. D. Mangus, G. A. Rodney, R. R. Shannon, and C. P. Spoelhof. The Hubble Space Telescope Optical Systems Failure Report. Techni- cal Report NASA-TM-103443, National Aeronautics and Space Administration, 1990.

[3] C. Bolkcom. V-22 Osprey Tilt-Rotor Aircraft: Congressional Research Service Report for Congress. Technical report, Congressional Research Service, Jan. 7, 2005.

[4] M. Hiltzik. 787 Dreamliner Teaches Boeing Costly Lesson on Outsourcing. Los Angeles Times, Feb. 15, 2011.

[5] National Aeronautics and Space Administration, Washington, DC. NASA Sys- tems Engineering Handbook, NASA/SP-2007-6105 Rev1 edition, 2007.

[6] N. P. Suh. The Principles of Design. Oxford University Press, New York, NY, 1990.

[7] J. J. Deyst. The Application of Estimation Theory to Managing Risk in Product Developments. In Proceedings of the 21st Digital Avionics Systems Conference, volume 1, pages 4A3–1–12, Irvine, CA, 2002.

[8] D. L. Allaire. Uncertainty Assessment of Complex Models with Application to Aviation Environmental Systems. PhD thesis, Massachusetts Institute of Technology, Cambridge, MA, 2009.

[9] A. Haldar and S. Mahadevan. Probability, Reliability and Statistical Methods in Engineering Design. John Wiley & Sons, Inc., New York, NY, 2000.

[10] C. J. Roy and W. L. Oberkampf. A Comprehensive Framework for Verification, Validation, and Uncertainty Quantification in Scientific Computing. Computer Methods in Applied Mechanics and Engineering, 200:2131–2144, 2011.

161 [11] R. C. Smith. Uncertainty Quantiﬁcation: Theory, Implementation, and Appli- cations. SIAM Computational Science & Engineering, Philadelphia, PA, 2014.

[12] W. L. Oberkampf, S. M. DeLand, B. M. Rutherford, K. V. Diegert, and K. F. Alvin. Error and Uncertainty in Modeling and Simulation. Reliability Engi- neering and System Safety, 75:333–357, 2002.

[13] A. H-S. Ang and W. H. Tang . Probability Concepts in Engineering: Emphasis on Applications to Civil and Environmental Engineering. John Wiley & Sons, Inc., Hoboken, NJ, 2nd edition, 2007.

[14] M. C. Kennedy and A. O’Hagan. Bayesian Calibration of Computer Models. Journal of the Royal Statistical Society, Series B, 63(3):425–464, 2001.

[15] B. M. Ayyub. Elicitation of Expert Opinions for Uncertainty and Risks. CRC Press, Boca Raton, FL, 2001.

[16] A. Babuscia. Statistical Risk Estimation for Communication System Design. PhD thesis, Massachusetts Institute of Technology, Cambridge, MA, 2012.

[17] A. Babuscia an K. Cheung. A Quantitative Approach to Perform Expert Elici- tation for Engineering Risk Analysis: Methodology and Experimental Results. Royal Statistical Society (Series A), 2013 (to appear).

[18] H. Agarwal, J. E. Renaud, E. L. Preston, and D. Padmanabhan. Uncertainty Quantiﬁcation Using Evidence Theory in Multidisciplinary Design Optimiza- tion. Reliability Engineering and System Safety, 85:281–294, 2004.

[19] J. C. Helton, J. D. Johnson, C. J. Sallaberry, and C. B. Storlie. Survey of Sampling-Based Methods for Uncertainty and Sensitivity Analysis. Reliability Engineering and System Safety, 91:1175–1209, 2006.

[20] I. M. Sobol’. On the Distribution of Points in a Cube and the Approximate Evaluation of Integrals. Zh. Vychisl. Mat. Mat. Fiz., 7(4):784–802, 1967.

[21] H. Niederreiter. Random Number Generation and Quasi-Monte Carlo Methods. Society for Industrial and Applied Mathematics, Philadelphia, PA, 1992.

[22] T. Bui-Thanh, K. Willcox, and O. Ghattas. Model Reduction for Large-Scale Systems with High-Dimensional Parametric Input Space. SIAM Journal on Scientiﬁc Computing, 30(6):3270–3288, 2008.

[23] C. Lieberman, K. Willcox, and O. Ghattas. Parameter and State Model Reduc- tion for Large-Scale Statistical Inverse Problems. SIAM Journal on Scientiﬁc Computing, 32(5):2523–2542, 2010.

162 [24] O. Bashir, K. Willcox, O. Ghattas, B. van Bloemen Waanders, and J. Hill. Hessian-Based Model Reduction for Large-Scale Systems with Initial Condition Inputs. International Journal for Numerical Methods in Engineering, 73(6):844– 868, 2008.

[25] D. Allaire and K. Willcox. Surrogate Modeling for Uncertainty Assessment with Application to Aviation Environmental System Models. AIAA Journal, 48(8):1791–1803, 2010.

[26] M. S. Eldred, A. A. Giunta, and S. S. Collis. Second-Order Corrections for Surrogate-Based Optimization with Model Hierarchies. In 10th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, AIAA Paper 2004- 4457, 2004.

[27] T. W. Simpson, J. D. Peplinski, P. N. Koch, and J. K. Allen. Metamodels for Computer-Based Engineering Design: Survey and Recommendations. En- gineering with Computers, 17(2):129–150, 2001.

[28] N. M. Alexandrov, R. M. Lewis, C. R. Gumbert, L. L. Green, and P. A. New- man. Approximation and Model Management in Aerodynamic Optimization with Variable-Fidelity Models. Journal of Aircraft, 38(6):1093–1101, 2001.

[29] T. D. Robinson, M. S. Eldred, K. E. Willcox, and R. Haimes. Surrogate-Based Optimization Using Multidelity Models with Variable Parameterization and Corrected Space Mapping. AIAA Journal, 46(11):2814–2822, 2008.

[30] R. Spence and R. S. Soin. Tolerance Design of Electronic Circuits. Imperial College Press, London, UK, 1997.

[31] K. Premphet and W. Paecharoenchai. Polypropylene/Metallocene Ethylene- Octene Copolymer Blends with a Bimodal Particle Size Distribution: Mechan- ical Properties and Their Controlling Factors. Journal of Applied Polymer Science, 85(11):2412–2418, 2002.

[32] M. E. Fowler, H. Keskkula, and D. R. Paul. Synergistic Toughening in Rubber Modiﬁed Blends. Polymer, 28(10):1703–1711, 1987.

[33] T. Usami, Y. Gotoh, and S. Takayama. Generation Mechanism of Short-Chain Branching Distribution in Linear Low-Density Polyethylenes. Macromolecules, 19(11):2722–2726, 1986.

[34] E. A. Murphy. One Cause? Many Causes?: The Argument from the Bimodal Distribution. Journal of Chronic Diseases, 17(4):301–324, 1964.

[35] P. Seeman, C. Ulpian, C. Bergeron, P. Riederer, K. Jellinger, E. Gabriel, G. P. Reynolds, and W. W. Tourtellotte. Bimodal Distribution of Dopamine Receptor Densities in Brains of Schizophrenics. Science, 225(4663):728–731, 1984.

163 [36] Y. Wu, C. Zhou, J. Xiao, J. Kurths, and H. J. Schellnhuber. Evidence for a Bimodal Distribution in Human Communication. Proceedings of the National Academy of Sciences, 107(44):18803–18808, 2010.

[37] D. G. Xenikos. Modeling Human Dialogue – The Case of Group Communica- tions in Trunked Mobile Telephony. Physica A, 388(23):4910–4918, 2009.

[38] D. Conway, E. Allison, R. Felstead, and M. Goulden. Rainfall Variability in East Africa: Implications for Natural Resources Management and Livelihoods. Philosophical Transactions of the Royal Society A, 363(1826):49–54, 2005.

[39] M. E. Angeles, J. E. Gonz´alez,N. D. Ram´ırez-Beltr´an,C. A. Tepley, and D. E. Comarazamy. Origins of the Caribbean Rainfall Bimodal Behavior. Journal of Geophysical Research: Atmospheres, 115(11):D11106, 2010.

[40] S. Touati, M. Naylor, and I. G. Main. Origin and Nonuniversality of the Earth- quake Interevent Time Distribution. Physical Review Letters, 102(16):168501, 2009.

[41] T. Parsons. Global Omori Law Decay of Triggered Earthquakes: Large After- shocks Outside the Classical Aftershock Zone. Journal of Geophysical Research: Solid Earth, 107(B9):ESE 9–1–20, 2002.

[42] K. v. Sichart and R.-P. Vollertsen. Bimodal Lifetime Distributions of Di- electrics for Integrated Circuits. Quality and Reliability Engineering Interna- tional, 7(4):299–305, 1991.

[43] K. L. Wong. The Physical Basis for the Roller-Coaster Hazard Rate Curve for Electronics. Quality and Reliability Engineering International, 7(6):489–495, 1991.

[44] F. Knoll and T. Vogel. Design for Robustness. IABSE (International Association for Bridge and Structural Engineering), Zurich, Switzerland, 2009.

[45] A. Karl. Robust Design and Stochastics: Key Methods, Case Studies and Challenges. In MDO Consortium Workshop, Stanford, CA, 2013.

[46] C. M. Creveling, J. L. Slutsky, and D. Antis, Jr. Design for Six Sigma in Technology and Product Development. Prentice Hall, Upper Saddle River, NJ, 2003.

[47] K. C. Kapur and L. R. Lamberson. Reliability in Engineering Design. John Wiley & Sons, Inc., New York, NY, 1977.

[48] L. A. Ealey. Quality By Design: Taguchi Methods and U.S. Industry. ASI Press, Burr Ridge, IL, 2nd edition, 2009.

164 [49] G. Taguchi. The System of Experimental Design: Engineering Methods to Op- timize Quality and Minimize Costs (2 vols.). UNIPUB/Kraus International Publications and American Supplier Institute, White Plains, NY and Dear- born, MI, 1987.

[50] M. S. Phadke. Quality Engineering Using Robust Design. Prentice Hall, Engle- wood Cliﬀs, NJ, 1989.

[51] R. A. Fisher. The Design of Experiments. Oliver & Boyd, Edinburgh, UK, 1935.

[52] G. E. P. Box, W. G. Hunter, and J. S. Hunter. Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building. John Wiley & Sons, Inc., New York, NY, 1978.

[53] W. J. Welch, T.-K. Yu, S. M. Kang, and J. Sacks. Computer Experiments for Quality Control by Parameter Design. Journal of Quality Technology, 22(1):15– 22, 1990.

[54] A. C. Shoemaker, K.-L. Tsui, and C. F. J. Wu. Economical Experimentation Methods for Robust Design. Technometrics, 33(4):415–427, 1991.

[55] R. H. Myers, A. I. Khuri, and G. Vining. Response Surface Alternatives to the Taguchi Robust Parameter Design Approach. The American Statistician, 46(2):131–139, 1992.

[56] D. D. Frey, F. Engenhardt, and E. M. Greitzer. A Role for One Factor a a Time Experimentation in Parameter Design. Research in Engineering Design, 14(2):65–74, 2003.

[57] D. D. Frey and H. Wang. Adaptive One-Factor-at-a-Time Experimentation and Expected Value of Improvement. Technometrics, 48(3):418–431, 2006.

[58] K.-L. Tsui. An Overview of Taguchi Method and Newly Developed Statistical Methods for Robust Design. IIE Transactions, 24(5):44–57, 1992.

[59] D. D. Frey and R. Jugulum. The Mechanisms by Which Adaptive One-Factor- at-a-Time Experimentation Leads to Improvement. Journal of Mechanical De- sign, 128(5):1050–1060, 2006.

[60] T. B. Savoie and D. D. Frey. Detecting Mistakes in Engineering Models: The Eﬀects of Experimental Eesign. Research in Engineering Design, 23(2):155–175, 2012.

[61] G. Eckes. The Six Sigma Revolution. John Wiley & Sons, Inc., New York, NY, 2001.

165 [62] G. Tennant. Six Sigma: SPC and TQM in Manufacturing and Services. Gower Publishing Company, Burlington, VT, 2001.

[63] A. M. Hasofer and N. C. Lind. Exact and Invariant Second-Moment Code Format. Journal of the Engineering Mechanics Division, 100(1):111–121, 1974.

[64] R. Rackwitz and B. Fiessler. Structural Reliability Under Combined Random Load Sequences. Computers and Structures, 9(5):489–494, 1978.

[65] B. Fiessler, H. J. Neumann, and R. Rackwitz. Quadratic Limit States in Struc- tural Reliability. Journal of Engineering Mechanics, 1095(4):661–676, 1979.

[66] H. O. Madsen, S. Krenk, and N. C. Lind. Methods of Structural Safety. Prentice Hall, Englewood Cliﬀs, NJ, 1986.

[67] W. Weaver. Science and Complexity. American Scientist, 36:536–544, 1948.

[68] B. A. Huberman and T. Hogg. Phase Transitions in Artiﬁcial Intelligence Sys- tems. Artiﬁcial Intelligence, 33:155–171, 1987.

[69] G. Johnson. Researchers on Complexity Ponder What It’s All About. The New York Times: Technology, May 6, 1997.

[70] S. Balestrini-Robinson. A Modeling Process to Understand Complex System Architectures. PhD thesis, Georgia Institute of Technology, Atlanta, GA, 2009.

[71] P. A. Corning. Complexity is Just a Word! Technological Forecasting and Social Change, 59:197–200, 1998.

[72] S. C. Bankes. Tools and Techniques for Developing Policies for Complex and Uncertain Systems. In Proceedings of the National Academy of Sciences of the United States of America, volume 99, pages 7263–7266, 2002.

[73] J. D. Summers and J. J. Shah. Mechanical Engineering Design Complex- ity Metrics: Size, Coupling, and Solvability. Journal of Mechanical Design, 132(2):021004–1–11, 2010.

[74] D. Braha and O. Maimon. The Measurement of a Design Structural and Func- tional Complexity. IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans, 28(4):527–535, 1998.

[75] A. J. Albrecht and Jr. J. E. Gaﬀney. Software Function, Source Lines of Code, and Development Eﬀort Prediction: A Software Science Validation. IEEE Transactions on Software Engineering, SE-9(6):639–648, 1983.

[76] P. A. Murmann. Expected Development Time Reductions in the German Me- chanical Engineering Industry. Journal of Product Innovation Management, 11(3):236–252, 1994.

166 [77] A. Griﬃn. The Eﬀect of Project and Process Characteristics on Product De- velopment Cycle Time. Journal of Marketing Research, 34(1):24–35, 1997.

[78] M. H. Meyer and J. M. Utterback. Product Development Cycle Time and Com- mercial Success. IEEE Transactions on Engineering Management, 42(4):297– 304, 1995.

[79] T. J. McCabe. A Complexity Measure. IEEE Transactions on Software Engi- neering, 2(4):308–320, 1976.

[80] M. V. Tatikonda and S. R. Rosenthal. Technology Novelty, Project Complex- ity, and Product Development Project Execution Success: A Deeper Look at Task Uncertainty in Product Innovation. IEEE Transactions on Engineering Management, 47(1):74–87, 2000.

[81] S. Novak and S. D. Eppinger. Sourcing by Design: Product Complexity and the Supply Chain. Management Science, 47(1):189–204, 2004.

[82] J. Kim and D. Wilemon. An Empirical Investigation of Complexity and Its Management in New Product Development. Technology Analysis & Strategic Management, 21(4):547–564, 2009.

[83] K. M. M. Hölttäand K. N. Otto. Incorporating Design Effort Complexity Measures in Product Architectural Design and Assessment. Design Studies, 26(5):463–485, 2005.

[84] J. Hartmanis and R. E. Stearns. On the Computational Complexity of Al- gorithms. Transactions of the American Mathematical Society, 117:285–306, 1965.

[85] S. Arora and B. Barak. Computational Complexity. Cambridge University Press, New York, NY, 2009.

[86] A. N. Kolmogorov. Three Approaches to the Quantitative Deﬁnition of Infor- mation. Problems of Information Transmission, 1(1):1–7, 1965.

[87] G. J. Chaitin. On the Simplicity and Speed of Programs for Computing Inﬁnite Sets of Natural Numbers. Journal of the ACM, 16(3):407–422, 1969.

[88] T. M. Cover and J. A. Thomas. Elements of Information Theory. John Wiley & Sons, Inc., New York, NY, 1991.

[89] C. A. Rodriguez-Toro, S. J. Tate, G. E. M. Jared, and K. G. Swift. Complexity Metrics for Design (Simplicity + Simplicity = Complexity). In Proceedings of the Institution of Mechanical Engineers, Part B: Journal of Engineering Man- ufacture, volume 217, pages 721–725, 2003.

167 [90] M. Tribus and E. C. McIrvine. Energy and Information (Thermodynamics and Information Theory). Scientiﬁc American, 225(3):179–188, 1971. [91] C. E. Shannon. A Mathematical Theory of Communication. The Bell System Technical Journal, 27:379–423 and 623–656, 1948. [92] J. Rifkin and T. Howard. Entropy: A New World View. The Viking Press, New York, NY, 1980. [93] J. S. Coleman. An Introduction to Mathematical Sociology. The Free Press, New York, NY, 1964. [94] K. D. Bailey. Sociological Entropy Theory: Toward a Statistical and Verbal Congruence. Quality and Quantity, 18(1):113–133, 1983. [95] K. D. Bailey. Social Entropy Theory. State University of New York Press, Albany, NY, 1990. [96] S. Lloyd and H. Pagels. Complexity as Thermodynamic Depth. Annals of Physics, 188:186–213, 1988. [97] D. Allaire, Q. He, J. Deyst, and K. Willcox. An Information-Theoretic Metric of System Complexity With Application to Engineering System Design. Journal of Mechanical Design, 134(10):100906–1–10, 2012. [98] N. P. Suh. Complexity: Theory and Applications. Oxford University Press, New York, NY, 2005. [99] L. L. Campbell. Exponential Entropy as a Measure of Extent of a Distribution. Probability Theory and Related Fields, 5(3):217–225, 1966. [100] G. H. Hardy, J. E. Littlewood, and G. P´olya. Inequalities. Cambridge University Press, Cambridge, UK, 1934. [101] H. V. Dannon. Power Means Calculus, Product Calculus, Harmonic Mean Calculus, and Quadratic Mean Calculus. Gauge Institute Journal, 4(4):1–88, 2008. [102] E. T. Jaynes. Information Theory and Statistical Mechanics. The Physical Review, 106(4):620–630, 1957. [103] E. T. Jaynes. Information Theory and Statistical Mechanics II. The Physical Review, 108(2):171–190, 1957. [104] B. M. Ayyub and G. J. Klir. Uncertainty Modeling and Analysis in Engineering and the Sciences. CRC Press, Boca Raton, FL, 2010. [105] M. H. DeGroot and M. J. Schervish. Probability and Statistics. Addison-Wesley, Boston, MA, 3rd edition, 2002.

168 [106] D. P. Bertsekas and J. N. Tsitsiklis. Introduction to Probability. Athena Scien- tiﬁc, Nashua, NH, 2nd edition, 2008.

[107] D. Mukherjee and M. V. Ratnaparkhi. On the Functional Relationship Between Entropy and Variance with Related Applications. Communications in Statistics – Theory and Methods, 15(1):291–311, 1986.

[108] N. Ebrahimi, E. Maasoumi, and E. S. Sooﬁ. Ordering Univariate Distributions by Entropy and Variance. Journal of Econometrics, 90(2):317–336, 1999.

[109] S. Kullback and R. A. Leibler. On Information and Suﬃciency. The Annals of Mathematical Statistics, 22(1):79–86, 1951.

[110] S. Verd´uand D. Guo. A Simple Proof of the Entropy-Power Inequality. IEEE Transactions on Information Theory, 52(5):2165–2166, 2006.

[111] S. Kaplan and B. J. Garrick. On the Quantitative Deﬁnition of Risk. Risk Analysis, 1(1):11–27, 1981.

[112] L. A. Cox, Jr. What’s Wrong with Risk Matrices? Risk Analysis, 28(2):497–512, 2008.

[113] J. Li and D. Xiu. Evaluation of Failure Probability Via Surrogate Models. Journal of Computational Physics, 229(23):8966–8980, 2010.

[114] A. Saltelli, K. Chan, and E. M. Scott. Sensitivity Analysis. John Wiley & Sons, Inc., New York, NY, 2000.

[115] R. I. Cukier, C. M. Fortuin, K. E. Shuler, A. G. Petschek, and J. H. Schaibly. Study of the Sensitivity of Coupled Reaction Systems to Uncertainties in Rate Coeﬃcients I: Theory. Journal of Chemical Physics, 59:3873–3878, 1973.

[116] J. H. Schaibly and K. E. Shuler. Study of the Sensitivity of Coupled Reac- tion Systems to Uncertainties in Rate Coeﬃcients II: Applications. Journal of Chemical Physics, 59:3879–3888, 1973.

[117] A. Saltelli and R. Bolado. An Alternative Way to Compute Fourier Amplitude Sensitivity Test. Computation Statistics & Data Analysis, 26:445–460, 1998.

[118] T. Homma and A. Saltelli. Importance Measures in Global Sensitivity Analysis of Nonlinear Models. Reliability Engineering & System Safety, 52(1):1–17, 1996.

[119] I. M. Sobol’. Sensitivity Estimates for Nonlinear Mathematical Models. Math- ematical Modeling and Computational Experiment, 1(4):407–414, 1993.

[120] I. M. Sobol’. Global Sensitivity Indices for Nonlinear Mathematical Models and Their Monte Carlo Estimates. Mathematics and Computers in Simulation, 55:271–280, 2001.

169 [121] I. M. Sobol’. Theorems and Examples on High Dimensional Model Represen- tation. Reliability Engineering & System Safety, 79(2):187–193, 2003.

[122] A. Saltelli, S. Tarantola, F. Campolongo, and M. Ratto. Sensitivity Analysis in Practice: A Guide to Assessing Scientiﬁc Models. John Wiley & Sons, Inc., Hoboken, NJ, 2004.

[123] J. J. Lee, I. A. Waitz, B. Y. Kim, G. G. Fleming, L. Maurice, and C. A. Holsclaw. System for Assessing Aviation’s Global Emissions (SAGE), Part 2: Uncertainty Assessment. Transportation Research Part D: Transport and Environment, 12(6):381–395, 2007.

[124] K. Marais, S. P. Lukachko, M. Jun, A. Mahashabde, and I. A. Waitz. Assessing the Impact of Aviation on Climate. Meteorologische Zeitschrift, 17(2):157–172, 2008.

[125] D. Allaire and K. Willcox. A Variance-Based Sensitivity Index Function for Factor Prioritization. Reliability Engineering and System Safety, 107:107–114, 2012.

[126] H. Liu, W. Chen, and A. Sudjianto. Relative Entropy Based Method for Prob- abilistic Sensitivity Analysis in Engineering Design. Journal of Mechanical De- sign, 128(2):326–336, 2006.

[127] H. Takeda, P. Veerkamp, T. Tomiyama, and H. Yoshikawa. Modeling Design Processes. AI Magazine, 11(4):37–48, 1990.

[128] J. D. Irwin and R. M. Nelms. Basic Engineering Circuit Analysis. John Wiley & Sons, Inc., Hoboken, NJ, 8th edition, 2005.

[129] FAA Aerospace Forecast, Fiscal Years 2009-2025. Technical report, United States Federal Aviation Administration, 2009.

[130] B. Metz, O. R. Davidson, P. R. Bosch, R. Dave, and L. A. Meyer, editors. Contribution of Working Group III to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge University Press, New York, NY, 2007.

[131] A. Sch¨aferand I. Waitz. Air Transportation and the Environment. Transport Policy, 2014. http://dx.doi.org/10.1016/j.tranpol.2014.02.012.

[132] Results From a Survey of the Nation’s 50 Busiest Commercial Service Airports. Technical Report GAO/RCED-00-222, United States General Accounting Of- ﬁce, 2000.

[133] Y. Schipper. Environmental Costs in European Aviation. Transport Policy, 11(2):141–154, 2004.

170 [134] P. J. Wolfe, S. H. L. Yim, G. Lee, A. Ashok, S. R. H. Barrett, and I. A. Waitz. Near-airport Distribution of the Environmental Costs of Aviation. Transport Policy, 2014. http://dx.doi.org/10.1016/j.tranpol.2014.02.023.

[135] Annex 16: Environmental Protection Volume I: Aircraft Noise. Technical report, International Civil Aviation Organization, 2005.

[136] S. McGuire. Sleep Disturbance Due to Aircraft Noise. Technical report, Part- nership for AiR Transportation Noise and Emissions Reduction (PARTNER), Project 25, 2009.

[137] H. Swift. A Review of the Literature Related to Potential Health Eﬀects of Aircraft Noise. Technical report, Partnership for AiR Transportation Noise and Emissions Reduction (PARTNER), Project 19, 2009.

[138] I. Waitz, J. Townsend, J. Cutcher-Gershenfeld, E. Greitzer, and J. Kerrebrock. Aviation and the Environment: A National Vision Statement, Framework for Goals and Recommended Actions. Technical report, Report to the United States Congress, 2004.

[139] A. Mahashabde, P. Wolfe, A. Ashok, C. Dorbian, Q. He, A. Fan, S. Lukachko, A. Mozdzanowska, C. Wollersheim, S. R. H. Barrett, M. Locke, and I. A. Waitz. Assessing the Environmental Impacts of Aircraft Noise and Emissions. Progress in Aerospace Sciences, 47(1):15–52, 2011.

[140] I. Waitz, S. Lukachko, Y. Go, P. Hollingsworth, K. Harback, and F. Morser. Requirements Document for the Aviation Environmental Portfolio Management Tool. Technical Report PARTNER-COE-2006-001, Partnership for AiR Trans- portation Noise and Emissions Reduction (PARTNER), Project 19, 2006.

[141] Q. He. Development of an Income-Based Hedonic Monetization Model for the Assessment of Aviation-Related Noise Impacts. Master’s thesis, Massachusetts Institute of Technology, Cambridge, MA, 2010.

[142] Q. He, C. Wollersheim, M. Locke, and I. Waitz. Estimation of the Global Impacts of Aviation-Related Noise Using an Income-Based Approach. Transport Policy, 2014. http://dx.doi.org/10.1016/j.tranpol.2014.02.020.

[143] J. P. Nelson. Meta-Analysis of Airport Noise and Hedonic Property Values: Problems and Prospects. Journal of Transport Economics and Policy, 38(1):1– 28, 2004.

[144] J. P. Nelson and R. B. Palmquist. Review of APMT Beneﬁt Valuation Meth- ods and Suggestions for Modiﬁcations. Technical report, Prepared for the Mas- sachusetts Institute of Technology, 2008.

171 [145] Z. Wadud. A Systematic Review of Literature on the Valuation of Local En- vironmental Externalities of Aviation. Technical report, Omega: Aviation in a Sustainable World, Cambridge, UK. [146] C. Kish. An Estimate of the Global Impact of Commercial Aviation Noise. Master’s thesis, Massachusetts Institute of Technology, Cambridge, MA, 2008. [147] P. A. Diamond and J. A. Hausman. Contingent Valuation: Is Some Number Better Than No Number? Journal of Economic Perspectives, 8(4):45–64, 1994. [148] S. Navrud. The State-of-the-Art on Economic Valuation of Noise. Technical report, Final Report to the European Commission DG Environment, 2002. [149] Economic Assessment of the Noise Stringency Scenarios. Technical Report CAEP/9-IP/12, ICAO Committee for Aviation Environmental Protection, 2013. [150] K. D. Kryter. The Meaning and Measurement of Perceived Noise Level. Noise Control, 6(5):12–17, 1960. [151] Federal Aviation Regulations, Noise Standards: Aircraft Type and Airworthi- ness Certification. Technical Report 14 CFR Part 36, United States Federal Aviation Administration, 2009. [152] Report on the Environmental Portion of the Noise Stringency Assessment. Tech- nical Report CAEP/9-IP/20, ICAO Committee for Aviation Environmental Protection, 2013. [153] Environmental Tool Suite Frequently Asked Questions. http: //www.faa.gov/about/office_org/headquarters_offices/apl/research/ models/toolsfaq/. United States Federal Aviation Administration [Online Resource]; Accessed March 11, 2014. [154] Cost-Benefit Analysis of CAEP/9 Noise Stringency Options. Technical Re- port CAEP/9-IP/25, ICAO Committee for Aviation Environmental Protection, 2013. [155] P. J. Wolfe and I. A. Waitz. Aircraft Noise Certification Stringency: Benefits, Costs, and Environmental Co-benefits. Transportation Research Part D, 2014 (in review). [156] Local Area Personal Income. http://www.bea.gov/regional/. United States Department of Commerce Bureau of Economic Analysis [Online Resource]; Ac- cessed June 15, 2010. [157] R. Tam, P. Belobaba, K. R. Polenske, and I. Waitz. Assessment of Silent Aircraft-Enabled Regional Development and Airline Economics in the UK. In Proceedings of the 45th AIAA Aerospace Sciences Meeting and Exhibit, Reno, NV, 2007.

172