Deriving Monotonic Function Envelopes from Observations* Herbert Kay Lyle H. Ungar Department of Computer Sciences Department of Chemical Engineering University of Texas at Austin University of Pennsylvania Austin, TX 78712 Philadelphia, PA 19104 bert@cs .utexas.edu [email protected] .edu
Abstract function f(x; 0) and then using regression analysis to determine the values 0 such that f(x; 0) .., g(x) . work in qualitative physics involves con- Much Traditional regression methods require knowledge of structing models of physical systems using func- the form of the estimation function f. For instance, tional descriptions such as "flow monotonically in- we may know that body weight is linearly related to creases with pressure." Semiquantitative meth- amount of body fat. We may then determine that ods improve model precision by adding numeri- f(weight; B) = weight - B is an appropriate model. cal envelopes to these monotonic functions . Ad Neural network methods have been developed for cases hoc methods are normally used to determine these where no information about f is known . This may be envelopes. This paper describes a systematic the case if f models a complex process whose physics is a envelope a method for computing bounding of poorly understood . Such networks are known to be ca- stream multivariate monotonic function given a pable of representing any functional relationship given of data. The derived envelope is computed by a large enough network. In this paper, we consider the determining a simultaneous confidence band for case where some intermediate level of knowledge about a special neural network which is guaranteed to f is known. In particular, we are interested in cases produce only monotonic functions. By compos- where we have knowledge of the monotonicity of f in ing these envelopes, more complex systems can be terms of the signs of where E x . For example, simulated using semiquantitative methods. xk we might know that outflowa from a tank monotonically increases with tank pressure. This type of knowledge Introduction is prevalent in qualitative descriptions of systems, so it makes sense to take advantage of it. Scientists and engineers build models of continuous Since the estimate is based on a finite set of data, it systems to better understand and control them. Ide- is not possible for it to be exact. We therefore require ally, these models are constructed based on the the our estimate to have an associated confidence measure underlying physical properties of the system. Unfortu- which takes into account the uncertainty introduced by nately, real systems are seldom well enough understood the finite sample size. For our semiquantitative repre- to construct precise models based solely on a priori sentation, we are therefore interested in deriving an knowledge of physical laws. Therefore, process data is envelope that bounds all possible functions that could often used to estimate some portions of the model. have generated the datastream with some probability Techniques for estimating a functional relationship P. between an output variable y and the input vector This paper describes a method for estimating and x typically assume that there is some determinis- computing bounding envelopes for multivariate func- tic function g and some random variate e such that tions based on a set of data and knowledge of the y = g(x) + e, where c is a normally distributed, mean monotonicity of the functional relationship . First, zero random variable with variance o-2 that represents we describe our method for computing an estimate measurement error and stochastic variations . The es- f(x; 0) ;zz~ g(x) based on a neural network that is con- timate is computed by using a parameterized fitting strained to produce only monotonic functions. Second, we describe our method for computing a bounding en- 'This work has taken place in the Qualitative Reasoning Group at the Artificial Intelligence Laboratory, The Uni- velope, which is based on linearizing the estimation versity of Texas at Austin. Research of the Qualitative function and then using F-statistics to compute a si- Reasoning Group is supported in past by NSF grants IRI- multaneous confidence band . Third, we present several 8905494, IRI-8904454, and IRI-9017047, by NASA contract examples of function fitting and its use in semiquanti- NCC 2-760, and by the Jet Propulsion Laboratory. tative simulation using QSIM. Next we discuss related
work in neural networks and nonparametric analysis and finally we summarize the results and describe fu- ture work. Bounded monotonic functions are a key element of the semiquantitative representation used by simulators such as Q2 [Kuipers and Berleant, 1988], Q3 [Berleant and Kuipers, 1992], and Nsim [Kay and Kuipers, 1993] which predict behaviors from models that are incom- pletely specified . To date, the bounds for such func- tions have been derived in an ad hoc manner . The work described in this paper provides a systematic method for finding these functional bounds. It is par- ticularly appropriate for semiquantitative monitoring and diagnosis systems (such as MIMIC [Dvorak and Kuipers, 1989]) because process data is readily avail- Figure 1: A neural net-based function estimator . able in such applications. By combining our function This three-layer net computes the function bounding method with model abduction methods such (E' + w wO[nh+1,1]) . as MISQ [Richards et ad., 1992], we plan to construct ~~ Al [w.[.i,l]a i=, wi[i,j]xi i[n+l,i1)~ + a self-calibrating system which derives models directly from observations of the monitored process. Such a system will have the property that as more data is ac- at 11 . All nodes use sigmoidal basis functions and all quired from the process, the model and its predictions connections are weighted . In our notation, wili j] rep- will improve. resents the connection from input xi to hidden node j and woU,ll represents the connection from hidden node network represents the Computing the Estimate j to the output layer' . This function Computing the estimate of g requires that we make Q(S some assumptions about the nature of deterministic = + wo[nh+1,11) and stochastic portions of the model. We assume that nh n the relationship between y and x is y = g(x)+c where e s -_ WOLi,11 0 wi[ii]xi + wi[n+lj] is a normally distributed random variable with mean 0 j=1 L ~Ei=1 and variance Q' . Other assumptions, such as different noise probability distributions or multiplicative rather where v(x) is the sigmoidal function We can than additive noise coupling could be made. The above compute the weights by solving the nonlinear least model, however, is fairly general and it permits us to squares problem use powerful regression techniques for the computation of the estimate and its envelope . For situations where Win (yi - yi)z variance is not uniform, we can use variance stabiliza- i tion techniques to transform the problem so that it has where Pi = f(xi ; w) and w is a vector of all weights. a constant variance. Cybenko [Cybenko, 1989] and others have shown that In traditional regression analysis, the modeler sup- with a large enough number of hidden units, any con- plies a function f(x; B) together with a dataset to a tinuous function may be approximated by a network least-squares algorithm which determines the optimal of this form. values for 9 so that f (x; 9) :z~ g(x) . The estimated One drawback of using this estimation function is that it can overfit the given data. This results in the value of y is then y = f(x; 9) . In our case, however, estimate following the random variate e as well as the the only information available about f is the signs of deterministic part of the model which means that we its n partial derivatives where xk E x, so no ex- get a poor approximation of g3. We therefore reduce One way to work plicit equation for f can beaassumed . the scope of possible functions to include only mono- for f is to use a neural net without an explicit form tonic functions. To do this, note that if f is mono- function estimator. Figure 1 illustrates a network for tonically increasing in xk, then must be positive. determining y given a set of inputs x. The network a has three layers. The input layer contains one node for 'The bias terms permit the estimation function to shift each element xk and one bias node set to a constant the center of the sigmoid. value of 1 . The hidden layer consists of a set of nh 'The weight wi[n+l,i] represents the connection to the input variable as nodes which are connected to each input bias and the weight wo[nh+1,1] represents the connec- well as to the bias input. The output layer consists tion from the hidden layer bias node to the output node. of a single node which is connected to all the hidden 'Visually, the estimate will try to pass through every nodes as well as to another bias value which is fixed datapoint.
By constraining the weights, we can force this deriva- tive to positive for all x, insuring that the resulting function is monotonic. The derivative in question is of = v' (s + Wo[ axk .n+1,1]) -p n~ ( n p t woU,110` E(w=[i,j]xi) + Wi[k,j]~ j=1 i=1 Since the derivative of the sigmoid function is positive for all values of its domain, will be positive if p is positive . This will be the caseaifbl
min yi)2 w (ya - i where bx depends on x . The confidence interval is a point probability measure since it expresses the uncer- subject to d 1
fined by
f(X;*)fs pF(a ;p,n - p)IIvTRv -1 JJ
where V = Qv The linearization can be I Ov I . viewed as defining a plane which is tangent to the true expectation surface (which is not a linear subspace) at w. Assuming that w is close to the exact value for w, the linearization will hold near this point on the plane. Because the linearization is only an approximation, this estimate does not provide an exact 1-a confidence band for f. However, the result is approximately cor- Figure 3: Reducing the size of a confidence band using rect and depends on the degree of nonlinearity of the monotonicity information . If we know that the un- expectation surface [Bates and Watts, 1988]. derlying function is monotonically increasing, we can We use the LINPACK subroutine DQRDC to compute rule out the shaded areas of the figure as part of the the QR decomposition of V. This method performs the envelope. decomposition using pivoting so that Rv is both upper triangular and has diagonal elements whose magnitude decreases going down the diagonal. With this form, we can easily recognize the case where the linearized ma- trix V may not have full rank (i.e ., only q < p columns are linearly independent) . This in turn means that Rv has zeros in all diagonals past column q . In such cases, the product vT Rv -1 may not have a solution. To handle this problem, we simply reduce the size of Rv by using only the upper left q x q corner. We must then also reduce the size of v so that it does not con- tain partials with respect to wi where i > q. This is justified, since if V forms a p-dimensional basis for the linearized expectation surface, and its rank is only q, then the wi where i > q may be ignored since their value is arbitrary . Figure 4: 95% envelope (solid lines) for the linear func- The confidence band is designed to cover all possi- tion y = 0.5x + 5 (dashed line) . The estimator has 3 ble curves that fit the data with a probability of 1- a. hidden nodes. Sample variance is 4 .520 . With traditional regression, this confidence band is the most precise description we can achieve given the gen- eral form of f . Using monotonicity information, how- expands at the ends since there is less data there to ever, we can further refine our prediction to derive a constrain the estimate. tighter envelope by ruling out portions of the curves The second example is that of a quadratic function that could not be monotonic . Consider the confidence with noise of variance 4. This dataset demonstrates the band shown in Figure 3. If we know that the underly- effect of nonuniform sampling. In areas where there is ing function is monotonic we may remove the shaded little data, the envelope bulges outward to compensate. regions from the prediction since no monotonic func- Note also that the upper portion of the envelope is tion within the confidence band could pass through narrowest at the far end of the plot. This is due to these regions . Note that this final step could not be bias in the estimation function which assumes that the performed if we used point confidence intervals. curve will "heel over" just past the end of the dataset. For this reason, the computed envelope is valid only Examples over the range of the given data. We have applied our envelope method to noisy datasets In the third example, the envelope method is used whose underlying functions are linear, quadratic, to create a semiquantitative model which is then sim- square root, and exponential . In order to give a feel- ulated using QSIM to predict the behavior of a real ing for the form of the envelopes generated, we present physical system. To develop this example, we took several of these test cases . Each result is for a uni- a small plastic tank and drilled a hole in its bottom. variate monotonically increasing model function f(x) . Our goal was to construct a semiquantitative model Figure 4 shows the estimate and envelope for a set of that could be used to predict the level of fluid in the 100 samples drawn from the function y = 0.5x+5 with tank over time . The standard approach to construct- additive noise with variance 4 . Note that the envelope ing such a model in QSIM is to construct a qualitative
120
Figure 5: 95% envelope (solid lines) for the quadratic Figure 6 : 95% envelope for level vs flow data from a function y = x2 + 5 (dashed line). The estimator has tank. The estimator has two hidden nodes . Sample two hidden nodes. Sample variance is 4.227. variance is 0 .441 .
model and then augment it with numerical informa- tion about the monotonic functions and parameters . Normally, these numerical envelopes are ad hoc in that they are hand-derived by the modeler. By applying the envelope method to a data stream from the system, we can construct these envelopes in a more principled way. We require that the model prediction bound all real behaviors of the system (i.e., that the true behavior is within the predicted bounds) with probability 1 - a . Semiquantitative models are particularly good at this because they produce a bounded solution. We begin by asserting the qualitative part of the model - A' = -f(A), f E M+ which asserts that the derivative of amount (A) is a monotonic function of amount . Since Figure 7: 95% envelope for the amount vs level data. we are interested in level rather than amount, we also The measurement accuracy for this data was greater assert that g(L) = A, g E M+, i .e., that level (L) than for the level vs flow data, hence the estimate is and amount are monotonically related. The complete narrower . The estimator has two hidden nodes. Sam- qualitative model is thus ple variance is 0.003.
A'= -f(g(L)) f, g E M+ tion of the datastream length, by This qualitative description holds for a wide class of observing the phys- ical system for a longer time period, tanks . While it is unquestionably true for our tank, we would expect the envelopes to tighten further, thus yielding it is not specific enough to produce useful predictions. tighter predictions. To increase the precision of the prediction, we must further define f(A) and g(L). To determine these functions, we performed the following experiment . We Related Work filled the tank with water and allowed it to drain while This work is related to several approaches to function measuring flow (cc/sec), level (cm), and amount (cc). estimation using neural networks. Cybenko [Cybenko, Figures 6 and 7 show the experimental data together 1989] has shown that any continuous function can be with the 95% confidence envelopes. With this infor- represented with a neural net that is similar to one mation, we used the QSIM semiquantitative simula- that we use. With our method, we assume a priori tion techniques Q2 [Kuipers and Berleant, 1988] and that the true function is monotonic. This assumption Nsim [Kay and Kuipers, 1993] to produce a prediction . restricts our attention to a smaller class of functions Since QSIM already reasons with monotonic envelopes, which means that less data is needed to compute a there are no special techniques required to handle the reasonable estimate . Additionally, we compute a con- composed function f(g(L)) . fidence measure on our estimate in the form of an en- Figure 8 shows the results of the simulation together velope. with experimental data for time vs level . Since the The VI-NET method [Leonard et al., 1992] also uses width of the monotonic function envelopes are a func- a neural network for computing estimates of general
information and a stream of data. It improves over existing methods for function estimation by providing a simultaneous confidence band that encloses all func- tions that could have generated the datastream . Using monotonicity information provides several benefits to function estimation " Much less data is required to obtain a reasonably precise model. " When combined with simultaneous confidence bands, portions of the band can be eliminated, thus further improving model precision. While the class of monotonic functions may seem restrictive for modeling continuous systems, we can Figure 8: Prediction envelope for the tank model us- represent many non-monotonic functions as compo- ing the previous monotonic envelopes for f and g. Note sitions of monotonic ones. For example, in a system that sample measurements taken from the system are of two cascaded tanks, the level of water in the lower all contained within the envelope . The predictions of tank is typically a non-monotonic function of time. this model could be strengthened with increased mea- The flow into the lower tank, however, is the compo- surements which would serve to reduce the monotonic sition of a monotonically increasing function of the envelopes . amount of water in the upper tank and a monoton- ically decreasing function of the water in the lower tank. By using semiquantitative simulation meth- functions. It is notable in that it provides a confidence ods, we can represent this composition and thus sim- measure on its estimate and can determine when it is ulate systems with non-monotonic behaviors. being asked to extrapolate. By using radial basis func- The method is applicable for cases where sample tions instead of sigmoidal ones, it is able to handle dif- variance is fixed. Future work includes adding vari- ferent variances across the function. This is especially ance stabilization methods to handle cases where the useful in applications where there is no a priori infor- data has a non-constant variance . We also plan to add mation about g . Because it allows non-constant vari- the ability to impose further constraints such as stating ances, it would be difficult to get a VI-NET to return that the function must pass through zero. simultaneous confidence bands, which are important Our method for deriving bounds for monotonic func- since we wish to bound all curves that could have gen- tions plays a key role in the construction of semiquanti- erated the datastream . In contrast, our approach can tative models, especially for monitoring and diagnosis only handle fixed-variance problems, but since variance tasks where process data is readily available. Because will often track either x or y, we can make monotonic- the precision of the resulting envelopes is a function of ity assumptions about o-2 if it should prove necessary. Under such circumstances, variance stabilization tech- the amount of data used to compute them, our bound- niques [Draper and Smith, 1981] should prove useful ing method also provides a systematic method for shift- ing the precision of a semiquantitative model along a in fixing the variance. continuum from purely qualitative to exact . This work is also related to monotonic function esti- mation [Kruskal, 1964; Hellerstein, 1990], particularly the NIMF estimator [Hellerstein, 1990] which also de- Nomenclature termines envelopes for multivariate monotonic func- x The domain of the function (y = g(x)). tions . NIMF allows for a more general noise model n The dimension of x. (zero mean, symmetrically distributed) and uses a non- nh The number of hidden units in the network. parametric statistical method for determining point p The number of weights in the network confidence bounds. These bounds, however, are much (a total of (n + 2)nh + 1) . weaker than those derived with methods that first com- Wi[z,j] The weight from xs to hidden unit j. pute an estimate . This is not surprising, since we are wou,l] The weight from hidden unit j to the output . assuming more about the noise than NIMF does. Fi- w A vector of all weights . nally, NIMF produces point bounds only, and it is un- y The estimate of y. clear how these bounds could be made into confidence References bands of reasonable width. Bates, Douglas M. and Watts, Donald G. 1988. Non- linear Regression and Its Applications. John Wiley Discussion and Future Work and Sons, New York. This paper has described a method for computing en- Berleant, Daniel and Kuipers, Benjamin 1992. Com- velopes for functions described solely by monotonicity bined qualitative and numerical simulation with q3 .
122 In Faltings, Boi and Struss, Peter, editors 1992, Re- cent Advances in Qualitative Physics. MIT Press. Biegler, L. T. and Cuthrell, J . E. 1985. Improved infeasible path optimization for sequential modular simulators - II. the optimization algorithm. Comput- ers and Chemical Engineering 9(3) :257-267 . Biegler, L. T. 1985. Improved infeasible path op- timization for sequential modular simulators - I. the interface. Computers and Chemical Engineering 9(3) :245-256 . Cybenko, G. 1989. Approximation by superpositions of sigmoidal functions. Mathematics of Control, Sig- nals, and Systems 2:303-314 . Draper, Norman R. and Smith, H . 1981. Applied Re- gression Analysis (second edition) . John Wiley and Sons, New York. Dvorak, Daniel Louis and Kuipers, Benjamin 1989. Model-based monitoring of dynamic systems. In Pro- ceedings of the Eleventh International Joint Confer- ence on Artificial Intelligence . 1238-1243. Hellerstein, Joseph 1990 . Obtaining quantitative pre- dictions from monotone relationships . In Proceedings of the Eighth National Conference on Artificial Intel- ligence (AAAI-90). 388-394. Hertz, John; Krogh, Anders; and Palmer, Richard G. 1991 . Introduction to the Theory of Neural Computa- tion. Addison-Wesley, Redwood City. Kay, Herbert and Kuipers, Benjamin 1993. Numerical behavior envelopes for qualitative models . In Proceed- ings of the Eleventh National Conference on Artificial Intelligence (AAAI-93). to appear. Kruskal, J . B. 1964. Multidimensional scaling by op- timizing goodness of fit to a nonmetric hypothesis. Psychometrika 29(1) :1-27. Kuipers, Benjamin and Berleant, Daniel 1988. Using incomplete quantitative knowledge in qualitative rea- soning . In Proceedings of the Seventh National Con- ference on Artificial Intelligence. 324-329 . Leonard, J . A. ; Kramer, M. A . ; and Ungar, L. H. 1992 . Using radial basis functions to approximate a function and its error bounds . IEEE Transactions of Neural Networks 3(4) :624-627 . Richards, Bradley L. ; Kraan, Ina; and Kuipers, Ben- jamin 1992 . Automatic abduction of qualitative mod- els. In Proceedings of the Tenth National Conference on Artificial Intelligence (AAAI-92). 723-728.