<p>OPTIMIZATION-BASED APPROXIMATE DYNAMIC <br>PROGRAMMING </p><p>A Dissertation Presented by <br>MAREK PETRIK </p><p>Submitted to the Graduate School of the <br>University of Massachusetts Amherst in partial fulfillment of the requirements for the degree of </p><p>DOCTOR OF PHILOSOPHY <br>September 2010 <br>Department of Computer Science </p><ul style="display: flex;"><li style="flex:1">c</li><li style="flex:1">ꢀ Copyright by Marek Petrik 2010 </li></ul><p>All Rights Reserved </p><p>OPTIMIZATION-BASED APPROXIMATE DYNAMIC <br>PROGRAMMING </p><p>A Dissertation Presented by <br>MAREK PETRIK </p><p>Approved as to style and content by: Shlomo Zilberstein, Chair Andrew Barto, Member Sridhar Mahadevan, Member Ana Muriel, Member Ronald Parr, Member <br>Andrew Barto, Department Chair Department of Computer Science <br>To my parents Fedor and Mariana </p><p>ACKNOWLEDGMENTS </p><p>I want to thank the people who made my stay at UMass not only productive, but also very enjoyable. I am grateful to my advisor, Shlomo Zilberstein, for guiding and supporting me throughout the completion of this work. Shlomo’s thoughtful advice and probing questions greatly influenced both my thinking and research.&nbsp;His advice was essential not only in forming and refining many of the ideas described in this work, but also in assuring that I become a productive member of the research community.&nbsp;I hope that, one day, I will be able to become an advisor who is just as helpful and dedicated as he is. </p><p>The members of my dissertation committee were indispensable in forming and steering the topic of this dissertation.&nbsp;The class I took with Andrew Barto motivated me to probe the foundations of reinforcement learning, which became one of the foundations of this thesis. Sridhar Mahadevan’s exciting work on representation discovery led me to deepen my understanding and appreciate better approximate dynamic programming. I really appreciate the detailed comments and encouragement that Ron Parr provided on my research and thesis drafts. Ana Muriel helped me to better understand the connections between my research and applications in operations research. Coauthoring papers with Jeff Johns, Bruno Scherrer, and Gavin Taylor was a very stimulating and learning experience.&nbsp;My research was also influenced by interactions with many other researches.&nbsp;The conversations with Raghav Aras, Warren Powell, Scott Sanner, and Csaba Szepesvari were especially illuminating. This&nbsp;work was also supported by generous funding from the Air Force Office of Scientific Research. </p><p>Conversations with my lab mate Hala Mostafa made the long hours in the lab much more enjoyable. While&nbsp;our conversations often did not involve research, those that did, motivated me to think deeper about the foundations of my work. I also found sharing ideas with my fellow grad students Martin Allen, Chris Amato, Alan Carlin, Phil Kirlin, Akshat Kumar, Sven Seuken, Siddharth Srivastava, and Feng Wu helpful in understanding the broader research topics.&nbsp;My free time at UMass kept me sane thanks to many great friends that I found here. </p><p>Finally and most importantly, I want thank my family.&nbsp;They were supportive and helpful throughout the long years of my education.&nbsp;My mom’s loving kindness and my dad’s intense fascination with the world were especially important in forming my interests and work habits.&nbsp;My wife Jana has been an incredible source of support and motivation in both research and private life; her companionship made it all worthwhile. It was a great journey. </p><p>v</p><p>ABSTRACT <br>OPTIMIZATION-BASED APPROXIMATE DYNAMIC <br>PROGRAMMING </p><p>SEPTEMBER 2010 MAREK PETRIK <br>Mgr., UNIVERZITA KOMENSKEHO, BRATISLAVA, SLOVAKIA <br>M.Sc., UNIVERSITY OF MASSACHUSETTS AMHERST Ph.D., UNIVERSITY OF MASSACHUSETTS AMHERST </p><p>Directed by: Professor Shlomo Zilberstein <br>Reinforcement learning algorithms hold promise in many complex domains, such as resource management and planning under uncertainty.&nbsp;Most reinforcement learning algorithms are iterative — they successively approximate the solution based on a set of samples and features. Although these iterative algorithms can achieve impressive results in some domains, they are not sufficiently reliable for wide applicability; they often require extensive parameter tweaking to work well and provide only weak guarantees of solution quality. </p><p>Some of the most interesting reinforcement learning algorithms are based on approximate dynamic programming (ADP). ADP, also known as value function approximation, approximates the value of being in each state.&nbsp;This thesis presents new reliable algorithms for ADP that use optimization instead of iterative improvement.&nbsp;Because these optimization–based algorithms explicitly seek solutions with favorable properties, they are easy to analyze, offer much stronger guarantees than iterative algorithms, and have few or no parameters to tweak. In&nbsp;particular, we improve on approximate linear programming — an existing method — and derive approximate bilinear programming — a new robust approximate method. </p><p>The strong guarantees of optimization–based algorithms not only increase confidence in the solution quality, but also make it easier to combine the algorithms with other ADP components. The other components of ADP are samples and features used to approximate the value function.&nbsp;Relying on the simplified analysis of optimization–based methods, we derive new bounds on the error due to missing samples.&nbsp;These bounds are simpler, tighter, and more practical than the existing bounds for iterative algorithms and can be used to evaluate solution quality in practical settings.&nbsp;Finally, we propose homotopy methods that use </p><p>vi the sampling bounds to automatically select good approximation features for optimization– based algorithms.&nbsp;Automatic feature selection significantly increases the flexibility and applicability of the proposed ADP methods. </p><p>The methods presented in this thesis can potentially be used in many practical applications in artificial intelligence, operations research, and engineering.&nbsp;Our experimental results show that optimization–based methods may perform well on resource-management problems and standard benchmark problems and therefore represent an attractive alternative to traditional iterative methods. </p><p>vii </p><p>CONTENTS </p><p>Page </p><p>ACKNOWLEDGMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;v ABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;xii </p><p>CHAPTER </p><p>1. INTRODUCTION&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 <br>1.1 Planning&nbsp;Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 Challenges&nbsp;and Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.3 Outline&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 </p><p>PART I: FORMULATIONS </p><p>2. FRAMEWORK:&nbsp;APPROXIMATE DYNAMIC PROGRAMMING . . . . . . .9 </p><p>2.1 Framework&nbsp;and Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2 Model:&nbsp;Markov Decision Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.3 Value&nbsp;Functions and Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 </p><p>2.4 Approximately&nbsp;Solving Markov Decision Processes . . . . . . . . . . . . . . . . . . . . . . . . . 16 </p><p>2.5 Approximation&nbsp;Error: Online and Offline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.6 Contributions&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 </p><p>3. ITERATIVE&nbsp;VALUE FUNCTION APPROXIMATION . . . . . . . . . . . . . . . .&nbsp;25 </p><p>3.1 Basic&nbsp;Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.2 Bounds&nbsp;on Approximation Error&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 </p><p>3.3 Monotonous&nbsp;Approximation: Achieving Convergence&nbsp;. . . . . . . . . . . . . . . . . . . . . . . 30 </p><p>3.4 Contributions&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 viii </p><p>4. APPROXIMATE&nbsp;LINEAR PROGRAMMING: TRACTABLE BUT </p><p>LOOSE APPROXIMATION. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;33 <br>4.1 Formulation&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 4.2 Sample-based&nbsp;Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 4.3 Offline&nbsp;Error Bounds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 4.4 Practical&nbsp;Performance and Lower Bounds&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 4.5 Expanding&nbsp;Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 4.6 Relaxing&nbsp;Constraints. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 4.7 Empirical&nbsp;Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 4.8 Discussion&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 4.9 Contributions&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 </p><p>5. APPROXIMATE&nbsp;BILINEAR PROGRAMMING: TIGHT </p><p>APPROXIMATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;52 <br>5.1 Bilinear&nbsp;Program Formulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 5.2 Sampling&nbsp;Guarantees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 5.3 Solving&nbsp;Bilinear Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 5.4 Discussion&nbsp;and Related ADP Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 5.5 Empirical&nbsp;Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 5.6 Contributions&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 </p><p>PART II: ALGORITHMS </p><p>6. HOMOTOPY&nbsp;CONTINUATION METHOD FOR APPROXIMATE </p><p>LINEAR PROGRAMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;69 <br>6.1 Homotopy&nbsp;Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 6.2 Penalty-based&nbsp;Homotopy Algorithm&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 6.3 Efficient&nbsp;Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 6.4 Empirical&nbsp;Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 6.5 Discussion&nbsp;and Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 6.6 Contributions&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 </p><p>7. SOLVING&nbsp;APPROXIMATE BILINEAR PROGRAMS . . . . . . . . . . . . . . . . .&nbsp;82 </p><p>7.1 Solution&nbsp;Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 </p><p>7.2 General&nbsp;Mixed Integer Linear Program Formulation&nbsp;. . . . . . . . . . . . . . . . . . . . . . . 83 7.3 ABP-Specific&nbsp;Mixed Integer Linear Program Formulation&nbsp;. . . . . . . . . . . . . . . . . . . 85 </p><p>7.4 Homotopy&nbsp;Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 ix <br>7.5 Contributions&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 </p><p>8. SOLVING&nbsp;SMALL-DIMENSIONAL BILINEAR PROGRAMS . . . . . . . . .&nbsp;90 </p><p>8.1 Bilinear&nbsp;Program Formulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 8.2 Dimensionality&nbsp;Reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 8.3 Successive&nbsp;Approximation Algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 8.4 Online&nbsp;Error Bound&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 8.5 Advanced&nbsp;Pivot Point Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 8.6 Offline&nbsp;Bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 8.7 Contributions&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 </p><p>PART III: SAMPLING, FEATURE SELECTION, AND <br>SEARCH </p><p>9. SAMPLING&nbsp;BOUNDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;109 <br>9.1 Sampling&nbsp;In Value Function Approximation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 9.2 State&nbsp;Selection Error Bounds&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 9.3 Uniform&nbsp;Sampling Behavior&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 9.4 Transition&nbsp;Estimation Error&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 9.5 Implementation&nbsp;of the State Selection Bounds. . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 9.6 Discussion&nbsp;and Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 9.7 Empirical&nbsp;Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 9.8 Contributions&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 </p><p>10.FEATURE SELECTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;129 <br>10.1 Feature&nbsp;Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 10.2 Piecewise&nbsp;Linear Features&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 10.3 Selecting&nbsp;Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 10.4 Related&nbsp;Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 10.5 Empirical&nbsp;Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 10.6 Contributions&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 </p><p>11.HEURISTIC SEARCH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;140 <br>11.1 Introduction&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 11.2 Search&nbsp;Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 11.3 Learning&nbsp;Heuristic Functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 11.4 Feature&nbsp;Combination as a Linear Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 </p><p>x<br>11.5 Approximation&nbsp;Bounds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 11.6 Empirical&nbsp;Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 11.7 Contributions&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 </p><p>12.CONCLUSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;168 </p><p>PART: APPENDICES </p><p>APPENDICES </p><p>A. NOTATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;171 B. PROBLEM DESCRIPTIONS&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;173 C. PROOFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;178 </p><p>BIBLIOGRAPHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;238 </p><p>xi <br>Introduction (1) <br>Heuristic Search (11) </p><p>Framework (2) <br>ALP (4) <br>Iterative Methods (3) <br>ABP (5) <br>Sampling Bounds (9) <br>Homotopy ALP (6) <br>Solving ABP (7) </p><p>Features (10) <br>Small-Dim ABP (8) </p><p>Figure 1. Chapter Dependence </p><p>xii </p><p>CHAPTER 1 <br>INTRODUCTION </p><p>Planning is the process of creating a sequence of actions that achieve some desired goals. Typical planning problems are characterized by a structured state space, a set of possible actions, a description of the effects of each action, and an objective function.&nbsp;This thesis treats planning as an optimization problem, seeking to find plans that minimize the cost of reaching the goals or some other performance measure. </p><p>Automatic planning in large domains is one of the hallmarks of intelligence and is, as a result, a core research area of artificial intelligence.&nbsp;Although the development of truly general artificial intelligence is decades — perhaps centuries — away, problems in many practical domains can benefit tremendously from automatic planning. Being able to adaptively plan in an uncertain environment can significantly reduce costs, improve efficiency, and relieve human operators from many mundane tasks.&nbsp;The two planning applications below illustrate the utility of automated planning and the challenges involved. </p><p>Blood inventory management&nbsp;A blood bank aggregates a supply of blood and keeps an inventory to satisfy hospital demands. The hospital demands are stochastic and hard to predict precisely.&nbsp;In addition, blood ages when it is stored and cannot be kept longer than a few weeks. The decision maker must decide on blood-type substitutions that minimize the chance of future shortage. Because there is no precise model of blood demand, the solution must be based on historical data.&nbsp;Even with the available historical data, calculating the optimal blood-type substitution is a large stochastic problem.&nbsp;Figure 1.1 illustrates the blood inventory management problem and Section B.2 describes the formulation in more detail. </p><p>Hospital 1 <br>Blood Bank </p><p><strong>B</strong><br><strong>0</strong></p><p><strong>AB A</strong><br><strong>A</strong></p><p>Hospital 2 </p><p>Figure 1.1. Blood Inventory Management <br>1</p><p><em>Irrigation </em></p><p><strong>Electricity Price </strong></p><p><strong>Reservoir </strong></p><p><strong>Water Inflow </strong></p><p>Figure 1.2. Reservoir Management <br>Water reservoir management&nbsp;An operator needs to decide how much and when to discharge water from a river dam in order to maximize energy production, while satisfying irrigation and flood control requirements. The challenges in this domain are in some sense complementary to blood inventory management with fewer decision options but greater uncertainty in weather and energy prices.&nbsp;Figure 1.2 illustrates the reservoir management problem and Section B.3 describes the formulation in more detail. </p><p>Many practical planning problems are solved using domain-specific methods. This&nbsp;entails building specialized models, analyzing their properties, and developing specialized algorithms. For example, blood inventory and reservoir management could be solved using the standard theory of inventory management (Axsater, 2006).&nbsp;The drawback of specialized methods is their limited applicability. Applying them requires significant human effort and specialized domain knowledge.&nbsp;In addition, the domain can often change during the lifetime of the planning system. </p><p>Domain-specific methods may also be inapplicable if the domain does not clearly fit into an existing category. For example, because of the compatibility among blood types, blood inventory management does not fit well the standard inventory control framework.&nbsp;In reservoir management, the domain-specific methods also do not treat uncertainty satisfactorily, neither do they work easily with historical data (Forsund, 2009). </p><p>This thesis focuses on general planning methods — which are easy to apply to a variety of settings — as an alternative to domain–specific ones.&nbsp;Having general methods that can reliably solve a large variety of problems in many domains would enable widespread application of planning techniques.&nbsp;The pinnacle of automatic planning, therefore, is to develop algorithms that work reliably in many settings. </p><p>1.1 Planning&nbsp;Models </p><p>The purpose of planning models is to capture prior domain knowledge, which is crucial in developing appropriate algorithms. The models need to capture the simplifying properties of given domains and enable more general application of the algorithms. </p><p>A planning model must balance domain specificity with generality.&nbsp;Very domain-specific models, such as the traveling salesman problem, may be often easy to study but have limited applicability. In&nbsp;comparison, very general models, such as mixed integer linear programs, are harder to study and their solution methods are often inefficient. </p><p>2<br>Aside from generality, the suitability of a planning model depends on the specific properties of the planning domain.&nbsp;For example, many planning problems in engineering can be modeled using partial differential equations (Langtangen, 2003).&nbsp;Such models can capture faithfully continuous physical processes, such as rotating a satellite in the orbit.&nbsp;These models also lead to general, yet well-performing methods, which work impressively well in complex domains.&nbsp;It is, however, hard to model and address stochasticity using partial differential equations.&nbsp;Partial differential equations also cannot be easily used to model discontinuous processes with discrete states. </p><p>Markov decision process (MDP) — the model used in this thesis — is another common general planning model.&nbsp;The MDP is a very simple model; it models only states, actions, transitions, and associated rewards.&nbsp;MDPs focus on capturing the uncertainty in domains with complex state transitions.&nbsp;This thesis is concerned with maximizing the discounted infinite-horizon sum of the rewards.&nbsp;That means that the importance of future outcomes is discounted, but not irrelevant.&nbsp;Infinite-horizon discounted objectives are most useful in long-term maintenance domains, such as inventory management. </p>