ON LINEAR CONTROL OF DECENTRALIZED STOCHASTIC SYSTEMS by Steven Michael Barta B.S., Yale University 1973 S.M., Massachusetts Institute of Technology 1976 SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY July, 1978 Massachusetts Institute of Technology 1978 Signature redacted Signature of Author. Department of Electrical Engineering and omPuerScience, July 5, 1978 Signature redacted Certified by. Thesis Sf ervisor Accepted by............................................................... Chairman, Departmental Committee on Graduate Students 2 ON LINEAR CONTROL OF DECENTRALIZED STOCHASTIC SYSTEMS by Steven Michael Barta Submitted to the Department of Electrical Engineering and Computer Science on July 5, 1978 in partial fulfillment of the requirements for the degree of Doctor of Philosophy. Abstract A general decision and control model for stochastic linear systems with several decision-makers who act on the basis of different informa- tion is formulated and analyzed. Many important systems, such as those described by stochastic linear differential equations and by stochastic linear differential-delay equations, are special cases of the model. Problems with the classical information pattern or with nonclassical information patterns, such as the no sharing or delayed sharing of obser- vations, can be studied with the model. To develop the existence results and optimality conditions, the controls are restricted to be causal linear functions of the information. Then the different information patterns are defined in terms of restric- tions on the linear operators defining the control laws. Existence of solutions to the feedback equations of the model is proven for linear control laws. For quadratic cost functions, general necessary conditions are presented. These conditions are equivalent to a set of nonlinear integral equations. When control actions propagate no faster through the system than information, the optimality conditions are also sufficient and are equivalent to a set of linear integral equations. The operator approach leads to a straightforward derivation of a general separation theorem for classical information patterns and an interpretation of fixed- structure constraints. For a decentralized version of linear quadratic loss decision theory, a certainty equivalent type separation theorem is proven and an integral equation similar to the Wiener-Hopf equation is derived to characterize the solution. By generalizing the innovations method of classical 3 estimation theory, the integral equation is solved. By specializing the problem to systems described by finite-dimensional state equations, a recursive solution reminiscent of the Kalman-Bucy filter is derived. For problems with delayed sharing of information, it is shown the control may be written in a separated form as the sum of two functions of certain uncontrolled processes. One function is linear in the control operator which operates on the information possessed by all stations (common information), while the other is nonlinear in the operator for information possessed by each station and uncorrelated with the common information. In certain cases, this separation of the control leads to a separation of the optimization problem into a centralized problem based on the common information and a decentralized problem based on the information uncorrelated with the common information. Thesis Supervisor: Nils R. Sandell, Jr. Title: Associate Professor of System Science and Engineering v 0 -o m z --I Cl) 5 Acknowledgment It is a pleasure for me to thank Professor Nils Sandell, my thesis supervisor. Throughout the course of this work, he made many valuable suggestions. I am grateful to Professors Michael Athans and Yu-Chi Ho, my thesis readers, for their stimulating comments. I appreciate the helpful discussions I have had with my fellow graduate students Douglas Looze, Ronald Feigenblatt, Demos Teneketzis, Robert Washburn, and Joseph Wall. For typing the thesis, I thank Nancy Ivey; for drawing the figures, I thank Arthur Giordani. Their work is excellent. This research was conducted at the M.I.T. Electronic Systems Laboratory with partial support provided by the National Science Foundation under grant NSF/ENG-77-19971, by the Office of Naval Research under contract ONR/N00014-76-C-0346, and by the Department of Energy under contract ERDA-E(49-18)-2087. 6 TABLE OF CONTENTS Chapter Page I Introduction 8 1.1. Decentralized Mathematical Optimization 8 1.2. Summary of thesis 16 1.3. Contributions of thesis 20 Ii A Stochastic Linear System Model 21 2.1. The Model 21 2.2. Examples 25 2.3. Discrete-time Version of Model 35 III Existence Results and Optimality Conditions 37 for Linear Control Laws 3.1. The Operator Formulation 37 3.2. Existence 40 3.3. Optimization 43 3.4. Partially Nested Systems 56 IV Operator Derivations for some Classical 61 and Nonclassical Models 4.1. Optimization with the 61 Classical Information Pattern 4.2. Decentralized Fixed-Structure 74 Optimization V Certainty Equivalent Solutions by a 81 Decentralized Innovations Approach 5.1. Separation of Estimation and 81 Decision-making in a Team 5.2. Solution of the Estimation Problem 91 7 Chapter Page 5.3. The Decentralized Kalman-Bucy Problem 98 5.4. The Second-Guessing Paradox 104 VI Delayed Sharing Information Patterns 113 6.1. Separation of the Control Law 113 6.2. Application to Optimization 123 VII Conclusions and Suggestions 129 for Future Research 7.1. Summary and Conclusions 129 7.2. Areas for Future Research 133 Appendix 1 135 Appendix II 140 Appendix III 142 References 144 Biographical Note 149 8 CHAPTER I Introduction 1.1. Decentralized Mathematical Optimization Many physical, social, and biological phenomena may be described by mathematical models. One of the principal components of such a model is a system. A set of equations, such as a set of differential equations, characterizes the system. Often there is a decision-maker (or a set of decision-makers) associated with the system (see Figure 1.1). The deci- sions are defined in terms of mathematical functions which depend on the decision-maker's information about the system. The objective of the decision-maker is to choose the actions so as to minimize or maximize some mathematical function which depends on the decisions and on certain variables related to the system. When the decisions affect the system, the decision-making process is called feedback control. In addition, it is often assumed there are disturbances affecting the system; these disturbances are usually modeled as random variables or as stochastic processes. To see how a typical engineering problem is formulated as an optimi- zation problem for a mathematical model, consider the control of message flow at one node of a data communication network or the similar situation of control of traffic flow at an entrance ramp to a highway. Suppose each system is described by a state variable. For the node this variable is the number of messages in the queue waiting to enter the network, while for the entrance ramp the state variable is the number of vehicles waiting to enter the highway. It is reasonable to assume the state variables DISTURBANCE SYSTEM DECISION- MAKER IM At IP L m - m m m - m m m - m m e - m m m m m uJ Fig. 1.1. System and Decision-maker. 10 satisfy a set of differential equations with stochastic driving terms, where these driving terms represent random inflows of messages or vehicles. Furthermore, the effects of the rest of the network on the node are also modeled by the stochastic process driving terms. Let the control actions affect the rates at which messages or vehicles leave the queues. Then a possible performance objective is to keep the flow rates near some pre- specified nominal values while not allowing the queues to become too large. To translate this objective into a mathematical performance index, we form the sum of a nonnegative increasing function of the absolute value of the state variable and a nonnegative increasing function of the absolute value of the deviation of the flow rate from the nominal rate. Then the cost criterion to be minimized is the expected value of the time integral of the sum. The information available to the controller at any time in- stant is the past record of the state variable values, the past record of control inputs, and a priori knowledge of the model and the statistics of the random disturbances. The mathematical theory of optimal decision and control is concerned with determining necessary and sufficient conditions to characterize the decisions which minimize or maximize a given performance criterion subject to the constraints imposed by the system structure. By combining ideas from the fields of probability, stochastic processes, statistics, game theory, calculus of variations, and mathematical programming, researchers have developed powerful tools, such as stochastic dynamic programming and the maximum principle, to analyze certain types of mathematical -- - F -...- 11 models.1 Although in many cases the solutions given by the theory are difficult to compute, the theory provides a unifying conceptual basis for studying mathematical optimization and it provides insight which is valuable in designing suboptimal controllers. An important theme in optimization theory (and also in other areas of engineering) is the use of on-line measurements, i.e. observations taken while the system is operating, in the optimal controller. In deter- ministic