CORE Metadata, citation and similar papers at core.ac.uk Provided by University of Wisconsin-Milwaukee University of Wisconsin Milwaukee UWM Digital Commons Theses and Dissertations May 2018 Numerical Solution of Stochastic Control Problems Using the Finite Element Method Maritn Gerhard Vieten University of Wisconsin-Milwaukee Follow this and additional works at: https://dc.uwm.edu/etd Part of the Mathematics Commons Recommended Citation Vieten, Maritn Gerhard, "Numerical Solution of Stochastic Control Problems Using the Finite Element Method" (2018). Theses and Dissertations. 1941. https://dc.uwm.edu/etd/1941 This Dissertation is brought to you for free and open access by UWM Digital Commons. It has been accepted for inclusion in Theses and Dissertations by an authorized administrator of UWM Digital Commons. For more information, please contact [email protected]. Numerical Solution of Stochastic Control Problems Using the Finite Element Method by Martin G. Vieten A Dissertation Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in Mathematics at The University of Wisconsin-Milwaukee May 2018 ABSTRACT Numerical Solution of Stochastic Control Problems Using the Finite Element Method by Martin G. Vieten The University of Wisconsin-Milwaukee, 2018 Under the Supervision of Professor Richard H. Stockbridge Based on linear programming formulations for infinite horizon stochastic control problems, a numerical technique in fashion of the finite element method is developed. The convergence of the approximate scheme is shown and its performance is illustrated on multiple examples. This thesis begins with an introduction of stochastic optimal control and a review of the theory of the linear programming approach. The analysis of existence and uniqueness of solutions to the linear programming formulation for fixed controls represents the first contri- bution of this work. Then, an approximate scheme for the linear programming formulations is established. To this end, a novel discretization of the involved measures and constraints using finite dimensional function subspaces is introduced. Its convergence is proven using weak convergence of measures, and a detailed analysis of the approximate relaxed controls. The applicability of the established method is shown through a collection of examples from stochastic control. The considered examples include models with bounded or unbounded state space, models featuring continuous and singular control as well as discounted or long- term average cost criteria. Analyses of various model parameters are given, and in selected examples, the approximate solutions are compared to available analytic solutions. A sum- mary and an outlook on possible research directions is given. ii Thank You. Professor Richard Stockbridge for the invaluable guidance, advice and expertise he offered during the last four years. His support allowed me to fully thrive in researching and present- ing this interesting topic. Professors Bruce Wade, Lei Wang, and Chao Zhu for serving on my doctoral committee and offering their feedback from reading this thesis as well as their insights at various stages of this work. Professor Gerhard Dikta for his continued support past the Master's level, and together with Professor Martin Reißel, for preparing me with the right tools and skills to succeed at researching and writing a dissertation. Dr. Rebecca Bourn for proofreading this thesis, and providing her feedback as well. Martin G. Vieten iii Table of Contents Introduction 1 I.1 On Mathematical Control . .1 I.2 Motivation and Overview . .3 Stochastic Control and Mathematical Background 11 II.1 Stochastic Control Problems . 11 II.1.1 Models Using Stochastic Differential Equations . 11 II.1.2 Martingale Problems and Relaxed Formulations . 16 II.1.3 Linear Programming Formulations . 24 II.2 Existence and Uniqueness under a Fixed Control . 32 II.2.1 Example: Long-Term Average Problem with Singular Behavior given by a Jump and a Reflection . 35 II.2.2 Example: Discounted Infinite Horizon Problem with Singular Behavior Given by Two Reflections . 48 II.3 Weak Convergence of Measures . 62 II.4 Cubic Spline Interpolation and B-Splines . 64 Approximation Techniques 67 III.1 Infinite Time Horizon Problems with Bounded State Space . 67 III.1.1 Discretization of the Linear Program . 68 III.1.2 Computational Set-up of Discretized Linear Program . 77 III.1.3 Evaluation of Cost Criterion . 85 III.1.4 Basis Functions and Meshing . 89 III.2 Infinite Time Horizon Problems with Unbounded State Space . 92 Convergence Analysis 99 IV.1 Infinite Time Horizon Problems with Bounded State Space . 99 IV.1.1 Optimality of the l-bounded (n; 1)-dimensional problem . 103 IV.1.2 Optimality of the l-bounded (n; m)-dimensional problem . 112 IV.1.3 Accuracy of Evaluation . 134 IV.2 Infinite Time Horizon Problems with Unbounded State Space . 140 iv Numerical Examples 147 V.1 Infinite Time Horizon Problems with Bounded State Space . 147 V.1.1 The Simple Particle Problem without Cost of Control . 148 V.1.2 The Simple Particle Problem with Cost of Control . 154 V.1.3 The Modified Bounded Follower . 157 V.1.4 The Modified Bounded Follower with Variable Jump Size . 166 V.1.5 Optimal Harvesting in a Stochastic Logistic Growth Example . 168 V.2 Infinite Time Horizon Problems with Unbounded State Space . 175 V.2.1 Optimal Control of a Cox-Ingersol-Ross Model . 175 V.2.2 Optimal Asset Allocation Problems with Budget Constraints . 182 Conclusion 198 VI.1 Summary . 198 VI.2 Outlook . 199 Appendix 203 A Theoretic Aspects of Singular Stochastic Processes . 203 A.1 Jump Processes . 203 A.2 Reflection Processes . 205 B Additional Lemmas Regarding Existence And Uniqueness under a Fixed Control209 B.1 Long-term Average Cost Criterion . 209 B.2 Infinite Horizon Discounted Criterion . 212 C Analytic Solutions to Selected Control Problems . 215 C.1 Simple Particle Problem without Cost of Control . 215 C.2 Modified Bounded Follower . 217 D List of Abbreviation and Symbols . 223 Bibliography 223 Curriculum Vitae 228 v List of Figures III.1 Mesh example for k(1) = 2, k(2) = 2, k(3) = 3 and k(4) =3.......... 91 V.1 Computed control, simple particle problem without costs of control . 150 V.2 State space density, simple particle problem without costs of control . 150 V.3 Computed control, simple particle problem with costs of control, c = 0:01 . 156 V.4 State space density, simple particle problem with costs of control, c = 0:01 156 V.5 Computed control, simple particle problem with costs of control, c = 1 . 157 V.6 State space density, simple particle problem with costs of control, c = 1 . 157 V.7 Computed control, simple particle problem with costs of control, c = 6 . 158 V.8 State space density, simple particle problem with costs of control, c = 6 . 158 V.9 Computed optimal control, modified bounded follower, coarse grid . 160 V.10 State space density, modified bounded follower, coarse grid . 160 V.11 Computed control, modified bounded follower, fine grid . 160 V.12 State space density, modified bounded follower, fine grid . 160 V.13 Computed control, modified bounded follower, with costs of reflection . 165 V.14 State space density, modified bounded follower, with costs of reflection . 165 V.15 Computed control, modified bounded follower with variable jump size . 167 V.16 State space density, modified bounded follower with variable jump size . 167 vi V.17 Computed control, coarse grid, stochastic logistic growth . 171 V.18 State space density, fine grid, stochastic logistic growth . 171 V.19 Computed control, Cox-Ingersol-Ross model . 177 V.20 State space density, Cox-Ingersol-Ross model . 177 V.21 Zoom of state space density, Cox-Ingersol-Ross model . 178 V.22 Re-evaluated state space density, Cox-Ingersol-Ross model . 178 V.23 Average optimal control, Cox-Ingersol-Ross model . 179 V.24 State space density, Cox-Ingersol-Ross model . 179 V.25 Computed control, asset allocation problem under geometric Brownian motion185 V.26 State space density, asset allocation problem under geometric Brownian motion185 V.27 Computed control, asset allocation problem under Ornstein-Uhlenbeck process189 V.28 State space density, asset allocation problem under Ornstein-Uhlenbeck pro- cess........................................ 189 V.29 Computed control, asset allocation problem under Ornstein-Uhlenbeck pro- cess, abnormal control behavior . 194 V.30 State space density, asset allocation problem under Ornstein-Uhlenbeck pro- cess, abnormal control behavior . 194 vii List of Tables II.1 Form of the singular integral term for different boundary behavior . 33 V.1 Probabilities for optimal control, simple particle problem without costs of control . 149 V.2 Parameter configuration (left), results and errors (right), simple particle problem without costs of control . 150 V.3 Results for optimality criterion and density, simple particle problem without costs of control, varying discretization levels, no extra mesh point . 151 V.4 Results for weights of singular occupation measure, simple particle problem without costs of control, varying discretization levels, no extra mesh point . 152 V.5 Results for optimality criterion and density, simple particle problem without costs of control, varying discretization levels, one additional mesh point . 153 V.6 Results for weights of singular occupation measure, simple particle problem without costs of control, varying discretization levels, one additional mesh point....................................... 154 V.7 Results, simple particle problem without costs of control, increasing number of control values . 155 V.8 Configuration, modified bounded follower problem . 159 viii V.9 Results (1), modified bounded follower, varying discretization levels . 161 V.10 Results (2), modified bounded follower, varying discretization levels . 162 V.11 Results, modified bounded follower, varying jump costs . 164 V.12 Results, modified bounded follower, varying reflection costs . 166 V.13 Results, modified bounded follower with variable jump size . 168 V.14 Convergence, stochastic logistic growth, fixed reflection point . 170 V.15 Approximate solutions for varying drift rate b ................ 173 V.16 Approximate solutions for varying diffusion rate σ .............