STOCHASTIC CONTROL FOR ECONOMIC MODELS Second Edition Books by David Andrew Kendrick Programming Investment in the Process Industries Notes and Problems in Microeconomic Theory (with Peter Dixon and Samuel Bowles ) The Planning of Industrial Investment Programs (with Ardy Stoutjesdijk) The Planning of Investment Programs in the Steel Industry (with Alexander Meeraus and Jaime Alatorre) GAMS: A User’s Guide (with Anthony Brooke and Alexander Meeraus) Feedback: A New Framework for Macroeconomic Policy Models for Analyzing Comparative Advantage Handbook of Computational Economics (edited with Hans M. Amman and John Rust) STOCHASTIC CONTROL FOR ECONOMIC MODELS Second Edition David A. Kendrick The University of Texas Typeset by VTEX Ltd., Vilnius, Lithuania (Rimas Maliukevicius and Vytas Statulevicius) STOCHASTIC CONTROL FOR ECONOMIC MODELS Second Edition, Version 2.00 2002 Copyright for the First Edition ©1981 by McGraw-Hill, Inc. Copyright transferred to David Kendrick in 1999. David Andrew Kendrick Department of Economics The University of Texas Austin, Texas, U.S.A. [email protected] http://eco.utexas.edu/faculty/Kendrick To Gail Contents Preface iv Preface to Second Edition vii 1 Introduction 1 I Deterministic Control 3 2 Quadratic Linear Problems 4 2.1 Problem Statement . ...................... 5 2.2 Solution Method .......................... 10 3 General Nonlinear Models 19 3.1 Problem Statement . ...................... 20 3.2 Quadratic Linear Approximation Method . ........ 21 3.3 Gradient Methods .......................... 25 3.4 Special Problems .......................... 27 3.4.1 Accuracy and Roundoff Errors ............... 27 3.4.2 Large Model Size ...................... 27 3.4.3 Inequality Constraints on State Variables . ........ 28 4 Example of Deterministic Control 29 4.1 System Equations .......................... 29 4.2 The Criterion Function . ...................... 35 iii CONTENTS iv II Passive-Learning Stochastic Control 37 5 Additive Uncertainty 38 5.1 Uncertainty in economic problems . ............... 38 5.2 Methods of Modeling Uncertainty . ............... 39 5.3 Learning: Passive and Active . ............... 40 5.4 Additive Error Terms . ...................... 43 6 Multiplicative Uncertainty 45 6.1 Statement of the Problem ...................... 45 6.2 Period Æ .............................. 47 ½ 6.3 Period Æ ............................ 50 6.4 Period ............................... 54 6.5 Expected Values of Matrix Products . ............... 55 6.6 Methods of Passive-Learning Stochastic Control . ........ 56 7 Example of Passive-Learning Control 57 7.1 The Problem . .......................... 57 7.2 The Optimal Control for Period 0 . ............... 58 7.3 Projections of Means and Covariances to Period 1 . ........ 63 III Active-Learning Stochastic Control 68 8 Overview 70 8.1 Problem Statement . ...................... 72 8.2 The Monte Carlo Procedure . ............... 75 8.3 The Adaptive-Control Problem: Initiation . ........ 76 8.4 Search for the Optimal Control in Period ............. 76 8.5 The Update . .......................... 80 8.6 Other Algorithms .......................... 81 9 Nonlinear Active-Learning Control 83 9.1 Introduction . .......................... 83 9.2 Problem Statement . ...................... 83 9.3 Dynamic Programming Problem and Search Method . 85 9.4 Computing the Approximate Cost-to-Go . ........ 85 9.5 Obtaining a Deterministic Approximation for the Cost-to-Go . 95 9.6 Projection of Covariance Matrices . ............... 97 CONTENTS v 9.7 Summary of the Search for the Optimal Control in Period ....101 9.8 Updating the Covariance Matrix . ...............102 9.9 Summary of the Algorithm . ...............102 10 Quadratic Linear Active-Learning Control 103 10.1 Introduction . ..........................103 10.2 Problem Statement . ......................103 10.2.1 Original System ......................103 10.2.2 Augmented System . ...............105 10.3 The Approximate Optimal Cost-to-Go ...............106 10.4 Dual-Control Algorithm ......................112 10.4.1 Initialization . ......................115 10.4.2 Search for the Optimal Control ...............115 10.5 Updating State and Parameter Estimates . ........118 11 Example: The MacRae Problem 120 11.1 Introduction . ..........................120 11.2 Problem Statement: MacRae Problem ...............120 11.3 Calculation of the Cost-To-Go . ...............122 11.3.1 Initialization . ......................122 11.3.2 Search for Optimal Control . ...............123 11.4 The Search . ..........................129 12 Example: Model with Measurement Error 133 12.1 Introduction . ..........................133 12.2 The Model and Data . ......................134 12.3 Adaptive Versus Certainty-Equivalence Policies . ........138 12.4 Results from a Single Monte Carlo Run . ........139 12.4.1 Time Paths of Variables and of Parameter Estimates . 141 12.4.2 Decomposition of the Cost-to-Go . ........154 12.5 Summary . ..........................164 Appendices 166 A Second-Order Expansion of System Eqs 166 CONTENTS vi B Expected Value of Matrix Products 169 B.1 The Expected Value of a Quadratic Form . ........169 B.2 The Expected Value of a MatrixTriple Product . ........170 C Equivalence of Matrix Riccati Recursions 172 D Second-Order Kalman Filter 176 E Alternate Forms of Cost-to-Go Expression 184 F Expectation of Prod of Quadratic Forms 187 F.1 Fourth Moment of Normal Distribution: Scalar Case . 188 F.2 Fourth Moment of Normal Distribution: Vector Case . 189 F.3 Proof . ..........................199 G Certainty-Equivalence Cost-To-Go Problem 203 H Matrix Recursions for Augmented System 206 I Vector Recursions for Augmented System 216 J Proof That Term in Cost-To-Go is Zero 221 K Updating the Augmented State Covariance 224 L Deriv of Sys Equations wrt Parameters 228 M Projection of the Augmented State Vector 232 N Updating the Augmented State Vector 236 O Sequential Certainty-Equiv Method 238 P The Reestimation Method 240 Q Components of the Cost-to-Go 242 R The Measurement-Error Covariance 245 S Data for Deterministic Problem 249 CONTENTS vii T Solution to Measurement Error Model 253 T.1 Random Elements ..........................253 T.2 Results . ..........................256 U Changes in the Second Edition 263 Bibliography 264 Preface This book is about mathematical methods for optimization of dynamic stochastic system and about the application of these methods to economic problems. Most economic problems are dynamic. The economists who analyze these problems study the current state of an economic system and ask how various policies can be used to move the system from its present status to a future more desirable state. The problem may be a macroeconomic one in which the state of the economic systems is described with levels of unemployment and inflation and the instruments are fiscal and monetary policy. It may be a microeconomic problem in which the system is characterized by inventory, sales, and profit levels and the policy variables are investment, production, and prices. It may be an international commodity-stabilization problem in which the state variables are levels of export revenues and inventories and the control variables are buffer-stock sales or purchases. Most economic problems are stochastic. There is uncertainty about the present state of the system, uncertainty about the response of the system to policy measures, and uncertainty about future events. For example, in macroeconomics some time series are known to contain more noise than others. Also, policy makers are uncertain about the magnitude and timing of responses to changes in tax rates, government spending, and interest rates. In international commodity stabilization there is uncertainty about the effects of price changes on consumption. The methods presented in this book are tools to give the analyst a better understanding of dynamic systems under uncertainty. The book begins with deterministic dynamic systems and then adds various types of uncertainty until it encompasses dynamic systems with uncertainty about (1) the present state of the system, (2) the response of the system to policy measures, (3) the effects of unseen future events which can be modeled as additive errors, and (4) errors in measurement. In the beginning chapters, the book is more like a textbook, but in the closing chapters it is more like a monograph because there is a relatively viii PREFACE ix widespread agreement about methods of deterministic-model solution while there is still considerable doubt about which of a number of competing methods of stochastic control will prove to be superior. As a textbook, this book provides a detailed derivation of the main results in deterministic and stochastic control theory. It does this along with numerical examples of each kind of analysis so that one can see exactly how the solutions to such models are obtained on computers. Moreover, it provides the economist or management scientist with an introduction to the kind of notation and mathematics which is used in the copious engineering literature on the subject of control theory, making access to that literature easier. Finally, it rederives some of the results in the engineering literature with the explicit inclusion of the kinds of terms typical of economic models. As a monograph, this book reports on a project explicitly designed to transfer some of the methodology of control theory from engineers to economists and to apply that methodology to economic problems to see whether it sheds