The Open Cybernetics and Systemics Journal, 2008, 2, 39-56 39 Sensitivity Analysis, a Powerful System Validation Technique Eric D. Smith, Ferenc Szidarovszky, William J. Karnavas and A. Terry Bahill* Systems and Industrial Engineering, University of Arizona, Tucson, AZ 85721-0020, USA Abstract: A sensitivity analysis is a powerful technique for understanding systems. This heuristic paper shows how to overcome some of the difficulties of performing sensitivity analyses. It draws examples from a broad range of fields: bioengineering, process control, decision making and system design. In particular, it examines sensitivity analyses of tradeoff studies. This paper generalizes the important points that can be extracted from the literature covering diverse fields and long time spans. Sensitivity analyses are particularly helpful for modeling systems with uncertainty. Keywords: Modeling, sensitivity, tradeoff studies, trade studies, verification, validation. 1. INTRODUCTION models [4], engineering models [5] and physiological models [6-9], to target disease treatment [10], in numerical computa- You should perform a sensitivity analysis anytime you tions [11], expert systems [12-14], discrete event simulations create a model, write a set of requirements, design a system, [15], where the techniques are called response surface meth- make a decision, do a tradeoff study, originate a risk analysis odology [16, 17], frequency domain experiments [18-20], or want to discover cost drivers. In a sensitivity analysis, you and perturbation analysis [21, 22]. When changes in the pa- change values of parameters or inputs (or architectural fea- rameters cause discontinuous changes in system properties, tures) and measure changes in outputs or performance indi- the sensitivity analysis is called that of singular perturbations ces. The results of a sensitivity analysis can be used to [23]. Sensitivity functions are used in adaptive control sys- validate a model, warn of unrealistic model behavior, point tems [24, 25]. In linear programming problems sensitivities out important assumptions, help formulate model structure, are referred to as marginal values or the shadow price of simplify a model, suggest new experiments, guide future each constraint and are interpreted as the increase in the op- data collection efforts, suggest accuracy for calculating pa- timal solution obtainable by each additional unit added to the rameters, adjust numerical values of parameters, choose an constraint [26]. operating point, allocate resources, detect critical criteria, suggest tolerance for manufacturing parts and identify cost Statistical techniques can be used when the model is so drivers. complex that computing the performance indices is very dif- ficult. For such complex models, the inputs are described A sensitivity analysis tells which parameters are the most with probability distributions and then the induced uncer- important and most likely to affect system behavior and/or tainly in the performance indices is analyzed: this approach predictions of the model. Following a sensitivity analysis, is known as probabilistic sensitivity analysis [27]. values of critical parameters can be refined while parameters that have little effect can be simplified or ignored. In the There is a large literature on global sensitivity analysis manufacturing environment, they can be used to allocate and regression/correlation techniques for large computer resources to critical parts allowing casual treatment of less simulation models [17]. We will not deal with such complex sensitive parts. If the sensitivity coefficients are calculated as simulations in this paper. The purpose of this paper is to pre- functions of time, it can be seen when each parameter has the sent simple techniques that will increase intuition and under- greatest effect on the output function of interest. This can be standing of systems. used to adjust numerical values for the parameters. The val- There are many common ways to do sensitivity analyses. ues of the parameters should be chosen to match the physical A partial derivative can be a sensitivity function for a system data at the times when they have the most effect on the out- described by equations. Otherwise, spreadsheets, like Excel, put. are convenient for doing sensitivity analyses of systems that The earliest sensitivity analyses that we have found are are not described by equations. This paper explores many the genetics studies on the pea reported by Gregor Mendel in techniques for doing sensitivity analyses. 1865 [1] and the statistics studies on the Irish hops crops by There are two classes of sensitivity functions: analytic Gosset writing under the pseudonym Student around 1890 and empirical. Analytic sensitivity functions are used when [2]. Since then sensitivity analyses have been used to analyze the system under study is relatively simple, well defined, and networks and feedback amplifiers [3], to validate social mathematically well behaved (e.g. continuous derivatives). These generally take the form of partial derivatives. They are *Address correspondence to this author at the Systems and Industrial Engi- convenient, because once derived they can often be used in a neering, University of Arizona, Tucson, AZ 85721-0020, USA; Tel: (520) broad range of similar systems and are easily adjusted for 621-6561; Fax: (520) 621-6555; E-mail: [email protected] changes in all model parameters. They also have an advan- 1874-110X/08 2008 Bentham Science Publishers Ltd. 40 The Open Cybernetics and Systemics Journal, 2008, Volume 2 Smith et al. tage in that the sensitivity of a system to a given parameter is Suppose that the output z is a valuable commodity (per- given as a function of all other parameters, including time or haps a potion). What is the easiest way (the smallest change frequency (depending on the model of the system), and can in one of these eight operating parameters) to increase the be plotted as functions of these variables. quantity of z that is being produced? This sounds like a natu- ral problem for absolute-sensitivity functions. Empirical sensitivity functions are often just point evaluations of a system's sensitivity to a given parameter(s) z S z = = x2 = 1, when other parameters are at known, fixed values. Empirical A A 0 sensitivity functions can be estimated from the point sensi- NOP z tivity evaluations over a range of the values of the parame- S z = = y2 = 1, B B 0 ters. They are generally determined by observing the changes NOP in output of a computer simulation as model parameters are z S z = = x y = 1, varied from run to run. Their advantage is that they are often C C 0 0 simpler (or more feasible) than their analytic counterparts or NOP z can even be determined for an unmodeled physical system. If S z = = x = 1, D D 0 the physical system is all that is available, the system output NOP is monitored as the inputs and parameters are changed from z S z = = y = 1, their normal values. E 0 E NOP z 2. ANALYTIC SENSITIVITY FUNCTIONS S z = = 1, F F This section will explain three different analytic sensitiv- NOP z ity functions. They are all based on finding the partial de- S z = = 2A x + C y + D = 0, x x 0 0 0 0 0 rivative of a mathematical system model with respect to NOP some parameter. Short examples are then given for each par- z S z = = 2B y + C x + E = 0. ticular form of a sensitivity function. y y 0 0 0 0 0 NOP 2.1. The Absolute-Sensitivity Function Evaluated at (x , y ) = (1, 1) , the six coefficients are 0 0 The absolute sensitivity of the function F to variations equally sensitive: increase any coefficient by one unit and in the parameter is given by the output increases by one unit. Since the derivatives with respect to x and y are equal to zero, to analyze sensitivity F F with respect to these variables we need to use higher order S = partial derivatives, as will be shown later. NOP where NOP means the partial derivative is evaluated at the What about interactions? Could we do better by changing Normal Operating Point (NOP) where all the parameters two parameters at the same time? Interaction terms are char- have their normal values. Of course, the function F must be acterized by, for example, the partial derivative of z with respect to x containing y. The interactions can be bigger than differentiable with respect to . In this paper, the function F the first-order partial derivatives. You can see that the above may also be a function of other parameters such as time, fre- partial derivatives with respect to the coefficients contain the quency or temperature. Absolute-sensitivity functions are operating point coordinates and the partial derivatives with useful for calculating output errors due to parameter varia- respect to the operating point coordinates depend on the co- tions and for assessing the times at which a parameter has its efficients. Therefore, we should expect the interactions to be greatest or least effect. Absolute sensitivities are also used in significant. Of the 64 possible second-partial derivatives, adaptive control systems. only the following are nonzero. The following two examples show the use of absolute- 2 sensitivity functions. The first shows how to tailor the output z z S = = 2x = 2, of a process and the second shows how to see when a pa- x A x A 0 NOP rameter has its greatest effect. 2 z S z = = y = 1, Example 1. A Process Model xC x C 0 NOP 2 Assume that the function z z SxD = = 1, f (x, y) z Ax2 By2 Cxy Dx Ey F xD = = + + + + + NOP 2 z models a process, where x and y are the inputs (raw material) S z = = 2y = 2, and A to F are models of the system parameters. Let the yB yB 0 NOP normal operating point be 2 z (x , y ) = (1,1), A = 1, B = 2,C = 3, D = 5, E = 7, F = 8, S z = = x = 1, 0 0 0 0 0 0 0 0 yC yC 0 producing z = 2.