Multiperiod Optimization: Dynamic Programming vs. Optimal Control: Discussion

Richard E. Howitt

The title of this session pitting Dynamic vey of solution approaches to control prob- Programming against Control Theory is mis- lems may be found in Polak [1973], and some leading since dynamic programming (DP) is specific texts are Bryson and Ho, Canon et an integral part of the discipline of control al., Dyer and McReynolds, Jacobson and theory. However, it is timely to discuss the Mayne, Polak [1971]. In the context of ap- relative merits of DP and other empirical plied economics, Zilberman cites many stud- solution approaches to control problems in ies that most members of the profession agricultural economics, and equally impor- would not classify as trivial. His citations tantly, how control theory can be used to include, of course, studies using DP solution pose more useful and realistic problems. approaches. From this point in the paper I shall, like The applied economics literature has pre- Burt, use the term Dynamic Programming dominantly used the maximum principle in (DP) to mean the classic solution procedure the analysis of the theoretical problems of developed by Bellman. All the other solution dynamic economic systems. While there are approaches used to solve multiperiod prob- special cases in which the first order condi- lems, including Differential Dynamic Pro- tions can be derived using Calculus of Varia- gramming [Jacobson and Mayne], are termed tions, Dynamic Programming or the Max- Control Theory. imum principle (and shown to be the same), Burt takes the session title literally and the Maximum principle is both less restric- rises to defend and extend the long history of tive in the form of the controls and con- DP research and application. I agree with straints than calculus of variations, and easier him concerning the difficulty of teaching ap- to interpret than DP. The maximum princi- plied problem formulation, but like Zilber- ple has an overriding advantage for economic man, feel that lack of popularity of DP cannot problems in that it explicitly specifies the be attributed to lack of exposure of graduate intertemporal qualitative properties of the students. imputed value of state variables as costate In his section on "Obstacles to Implemen- variables. In addition, Lagrangian interpreta- tation" Burt is unjustifiably pessimistic in his tions for binding constraints are incorporated judgement of the practicality and theoretical in the first order conditions. basis of solutions based on the Pontryagin In the same section, Burt generalizes the Maximum principle. Given the long history major problem with DP - the "curse of of Pontryagin based control applications in dimensionality" to "an inherent characteris- engineering and operations research, the tic of dynamic optimization problems in gen- problems cannot be categorized in general as eral when the number of state variables is "trivial" or without "theorems on the struc- large (more than 3 or 4)." However, much ture of the solution." A comprehensive sur- larger problems are routinely solved by analytic DP or programming approaches. Richard E. Howitt is an associate professor in the De- partment of Agricultural Economics at the University of The difference between the dimensionality California, Davis. increases for DP and nonlinear programming 413 December 1982 Western Journal of Agricultural Economics solutions is substantial. The DP memory re- of econometric models to optimum control is quirement is the grid size raised to the power shown in Rausser and Hochman, and Chow. of the number of states, whereas the memory Burt provides a comprehensive review and requirements for a nonlinear programming extension of methods to reduce state vector problem are proportional to the square of the decisions. The wheat storage study [Burt, product of state and time period dimensions. Koo and Dudley] would make a good vehicle Clearly, the effect of adding additional state for study of alternative solution methods, dimensions is very different. Intrilligator being apparently solvable by the linear cites a very difficult DP problem of grid Quadratic Gaussian formulation for Differ- size/state of 100 and four states. Assuming 20 ential Dynamic Programming or nonlinear time periods and four controls with inequali- programming, without any simplification of ty constraints on all states and controls in all state varibles, a wider choice of controls, and time periods, the same problem would yield the simplification of a stochastic error term. a nonlinear programming problem with 160 Among the methods for reducing dimen- columns and 240 rows, a routine operation sionality in DP I would add that the regener- using modern algorithms. A problem with ation point approach [Dryfus and Law] has a nine states and thirty yearly time periods was natural application to problem of capacity solved as a quadratic program in Noel and expansion and capital replacement. In these Howitt. latter problems, the advantage that DP en- Burt characterizes the dilemma of the ap- joys in integer problems makes it a logical plied analyst in the statement that "the pri- choice. I am less convinced as to the value of mary objective in all modeling is to capture DP in analyzing farm firms where crop alter- the essential aspects of the phenomenon natives, cash flows, asset stocks and updated under study and yet keep the model as sim- rational expectations would probably expand ple as possible." The choice of solution ap- the state dimension to an unmanageable proach thus depends on the appropriate level for DP. model characteristics. I agree with Burt that Zilberman's paper is written from a more for low dimension problems DP is a superior general view point which sees dynamic pro- method, particularly if the functions are ir- gramming as an important part of the set of regular and a range of stochastic values is solution approaches to the general stochastic important to the problem. However, for control problem. Zilberman skirts the details problems in which the simultaneity of many of solution procedures and concentrates on states is more important than smooth approx- the values of optimal control theory in posing imations to irregular functions there are sev- theoretical empirical and policy models in an eral additional solution approaches. Further- extension of comparative statics. To his re- more given the structures of micro theory view of economic applications of control and limitations of many least squares and theory, I would add the areas of management maximum likelihood approaches, most science [Bensoussan, Kleindorfer and applied economic problems satisfy the Tapiero], consumer demand [Houthakker requirements of non-DP approaches. Specifi- and Taylor] and rational expectations [Tay- cally: continuous and convex or concave func- lor]. At the end of Zilberman's review of tions, differentiability, and stochastic prop- control applications to agricultural econom- erties that can be characterized by sufficient ics, he remarks correctly that "it is still a statistics. limited tool in its application and impacts." Considerable work has been done on the He attributes this outcome to the emphasis stochastic control problem since Kushner on solution techniques rather than policy and Schweppe's cited article, a more recent results, a crime of which the first part of this article with illustrative examples is found in paper is guilty. However, as Burt points out, Haussman. The direct applicability of a range empirical problems have to be formulated 414 Howitt Discussion appropriately to be solvable, and I think it economics. The qualitative properties of equally important that practitioners are comparative dynamic equilibria are often not familiar with enough solution methodolgies easy to obtain in "nice" forms. An additional to minimize the "damage" to the theoretical source not mentioned by Zilberman or Burt problem. which works towards comparative dynamics The majority of Zilberman's paper de- is Kamien and Schwartz. Like Zilberman, I velops optimal control solutions to problems feel that there are valuable and relevant rela- in domains other than time. The results are tionships to be derived using control theory. stimulating and demonstrate the power of Burt emphasizes the empirical intricacies the maximum principle to extend micro- of solution by Bellman's original DP method, economics. In his two-stage optimal control however, the disparagment of other solution development over time and nontime do- methods is unnecessary. The fundamental mains, Zilberman develops the mi- concept of dynamic programming, the princi- croeconomic equivalent of Isard et al.'s speci- ple of optimality, has been used to develop a fication. Given the fixed costs of irrigation wide variety of solution procedures that are adoption, it is not clear that the combined not subject to the curse of dimensionality. problem can be always decoupled to allow The stage wise solution is maintained but the the two-stage optimization procedure sug- discretization of state and control space and gested. For instance, a major problem facing its attendant curse is avoided by storing func- irrigated agriculture is the decline in produc- tional forms that characterize the optimal tivity from rising water tables, salinity build- solution in the backward solution phase. This up or both. Evidently the distribution of land class of solution approaches is called differ- class is in this case a dynamic phenomena ential dynamic programming DDP. A which is affected by both the temporal alloca- through treatment of DDP may be found in tion of water and the adoption rates of irriga- Jacobson and Mayne (1970), and converg- tion technology. Zilberman's development is ence of the approximated problem to the true attractive as it lays out a microeconomic foun- nonlinear problem has been shown to be dation for rational technological change. quadratic. Throughout Zilberman's exposition of time The simplest example of DDP is the wide- and nontime domain optimization I had mis- ly used Linear Quadratic Gaussian (LQG) givings on the data availability to use the procedure briefly referred to by Burt. This approaches. Zilberman is aware of these con- procedure while not without drawbacks, has straints, but is optimistic that they can be immediate appeal for many economic prob- ameliorated by the rapid diffusion of mini- lems where the system dynamics are es- computers and networks in the industry. He timated by several (usually greater than four) is probably correct, although compatability of linear relationships with additive normal the variable data sources is likely to be stochastic terms. The objective function is troublesome. often characterized as a weighed combination To return to the central direction of this of producer and consumer surplus, which session the two widely differing papers each given linear estimates of demand and supply point to problems in the evolution of any new over the relevant range, results in a quadratic applied methodology. The first priority is to objective function. An example of a similar emphasize Zilberman's point that optimal problem specification is found in Burt, Koo control must first offer additional mi- and Dudley. croeconomic insights over comparative stat- The problem of inequality constraints in ics. Once the value to policy questions of the DDP is illustrated by Murray and Yakowitz's additional time dimensions is clear, there application to the multireservoir control will be an incentive to extend the compara- problem. In many other cases inequality con- tive static models of traditional agricultural straints can be approximated by suboptimal 415 December 1982 Western Journal of Agricultural Economics truncation or penalty functions incorporated Box, M. J. "A New Method of Constrained Optimization in the LQG approach. and A Comparison with Other Methods," Computer Journal 8(1965):42-52. An alternative solution approach that has considerable appeal and future potential is Burt, O. R., W. W. Koo and N. J. Dudley. "Optimal the solution of deterministic control prob- Stochastic Control of U.S. Wheat Stocks and Ex- lems by nonlinear programming. Efficient ports," American Journal of Agricultural Economics general algorithms for sparse problems that 62(1980):173-87. are nonlinear in both the objective function Bryson, A. E. and Y. C. Ho. Applied Optimal Control, and the constraints are now generally availa- John Wiley, New York, 1975. ble in MINOS, Murtagh and Saunders or Box and can be used to solve substantial Canon, M. D., C. D. Cullum and E. Polak. Theory of empirical problems such as Richardson and Optimal Control and Mathematical Programming, McGraw-Hill, New York, 1970. Ray and Noel and Howitt. Further develop- ments to combine this approach with stoch- Chow, G. C. Analysis and Control of Dynamics astic simulations are likely to be forthcoming. Economics Systems, John Wiley, New York, 1975. Where the constraint set can be success- fully embedded in the objective function, Dryfus, S. E. and A. M. Law. The Art and Theory of very large control models can be solved by Dynamic Programming, Academic Press, New York, 1977. unconstrained optimization methods. Fair discusses methods for solving up to 100 state Dyer, P. and S. R. McReynolds. The Computation and variable models and uses a nineteen state Theory of Optimal Control, Academic Press, New twenty time period model to compare alter- York, 1970. native approaches. Methods to extend this Fair, R. C. "On the Solution of Optimal Control Prob- type of solution to stochastic problems are lems as Maximization Problems," Annals of Econom- developed in Tinsley, Craine and Havenner ics and Social Measurement, 311(1974):135-54. and are currently implemented (but not pub- lished) on the four hundred equation Federal Haussman, U. G. "Some Examples of Optimal Stochast- Reserve model. ic Controls Or: The Stochastic Maximum Principle at Work," SIAM Review 23(1981):292-307. Inevitably there are trade-offs but no in- surmountable barriers in empirically imple- Houthakker, H. S. and L. D. Taylor. Consumer De- menting control problems, particularly in the mand in the United States: Analyses and Projections, form that agricultural economists have tradi- Harvard University Press, 1970. tionally specified them. What is now needed Intrilligator, M. D. Mathematical Optimization and from advocates of control theory (that of Economic Theory, Prentice Hall, 1971. course includes dynamic programming) is a clearer analytical exposition of the advan- Isard, W., P. Liossatos, Y. Kanemoto, and P. C. Kaniss. tages of comparative dynamics and dynamic Spatial Dynamics and Optimal Space-Time Develop- policy models over their static counterparts. ment, North Holland, New York, 1979. Jacobson, D. H. and D. Q. Mayne. Differential Dynam- ic Programming, American Elsevier New York, 1970.

Kushner, H. J. and F. C. Schweppe. "A Maximum Prin- ciple for Stochastic Control Systems," J. Mathematical References Analysis and Applications 8(1964):287-302.

Bellman, R. Dynamic Programming, Princeton Univer- Kamien, M. I. and N. L Schwartz. Dynamic Optimza- sity Press, 1957. tion, North Holland, New York, 1981.

Bensoussan, A., P. R. Kleindorfer and C. S. Tapiero. Applied Optimal Control, Studies in the Management Murray, D. M. and S. J. Yakowitz. "Constrained Differ- Sciences, Vol. 9 North Holland, Amsterdam, 1978. ential Dynamic Programming and its Application to 416 Howitt Discussion

Multireservoir Control," Water Resources Research, Rausser, G. C. and E. Hochman. Dynamic Agricultural 15(1979):1017-27. Systems: Economic Predictionand Control North Hol- land, New York, 1979. Murtagh, B. A. and M. A. Saunders. "Large Scale Linearly Constrained Optimization," Mathematical Richardson, J. W. and D. E. Ray. "Commodity Pro- Programming 14(1978):41-72. grams and Control Theory," American Journal of Agricultural Economics 64(1982):28-38. Noel, J. E. and R. E. Howitt. "Conjunctive Multibasin Management: An Optimal Control Approach," Water Taylor, J. B., Editor. "Issue on Rational Expectations Resources Research (forthcoming 1982). Analysis." Journal of Economic Dynamics and Control 2:1 February 1980. Polak, E. Computational Methods in Optimization Aca- demic Press, New York 1971. Tinsley, P., R. Craine, and A. Havenner. "Optimal Control of Large Nonlinear Stochastic Econometric . "An Historical Survey of Computational Meth- Models," Proceedings Summer Computer Simulation ods in Optimal Control," SIAM Review 15(1973):553- Conference Chicago, IL, IEEE, 1977. 84.

417