Mathematical Economics Deterministic dynamic optimization Continuous time Paulo Brito [email protected] 18.12.2017 Contents 1 Introduction 1 2 Calculus of variations 3 2.1 The simplest problems ................................... 6 2.1.1 Fixed end state and time ............................. 6 2.1.2 The simplest free terminal state problem ..................... 13 2.1.3 The free terminal time problem .......................... 16 2.2 Terminal constraints and scrap values .......................... 17 2.2.1 Equality terminal constraints ........................... 18 2.2.2 Inequality terminal constraint .......................... 20 2.3 Discounted infinite horizon problem ........................... 21 2.3.1 Free asymptotic state ............................... 22 2.3.2 Calculus of variations: discounted infinite horizon constrained terminal value 23 2.4 Canonical representation of CV problems ........................ 25 3 Optimal control 26 3.1 The simplest problems ................................... 27 3.1.1 The fixed terminal state problem ......................... 27 3.1.2 The free terminal state problem .......................... 29 3.1.3 Free terminal time ................................. 30 3.1.4 Examples ...................................... 31 1 Paulo Brito Mathematical Economics, 2018/19 0 3.2 Terminal constraints .................................... 35 3.2.1 Example ....................................... 36 3.3 Discounted infinite horizon problems ........................... 37 3.3.1 Examples ...................................... 39 3.4 The Hamiltonian dynamic system ............................. 42 3.5 Bibliographic references .................................. 46 4 Dynamic programming 48 4.1 Finite horizon ........................................ 48 4.2 Infinite horizon discounted problem ........................... 50 4.2.1 Solving the HJB equation explicitly ....................... 52 4.2.2 Examples ...................................... 52 4.3 References .......................................... 53 Appendices 55 A Proofs 56 A.1 Proofs of chapter 2 ..................................... 56 A.2 Proofs of chapter 3 ..................................... 59 A.3 Proofs of section 4 ..................................... 64 Chapter 1 Introduction This note deals with dynamic optimisation in continuous time, for one-dimensional problems. The state of a system is represented by function x : T ! R, where the domain of the independent variable is a subset of R+, T ⊆ R+, and x(t) 2 R, that is, we deal with scalar, or one-dimensional, problems. We represent the path or orbit of x by x = (x(t))t2T to distinguish it from the value taken at time t, x(t). ∗ ∗ Again, we want to find optimal paths, x = (x (t))t2T which solve problems of dynamic opti- mization problems. Two ypes of problems are considered 1. calculus of variations problems 2. optimal control problems Optimal control problems can be solved by one of the following methods 1. by using the Pontriyagin’s principle 2. by using the dynamic programming. Some optimal control problems can be converted into calculus of variations problems, although the reverse is not always possible. This means that calculus of variations can provide another method for solving optimal control problems. 1 Paulo Brito Mathematical Economics, 2018/19 2 ∗ ∗ The calculus of variations problem is : find a flow x ≡ (x (t))t2T, where T = [0;T ], or T = [0;T ), that maximises the value functional Z T V [x] = F (_x(t); x(t); t)dt + S(x(T );T ) 0 given the initial value x(0) = x0 and possibly some additional information on the terminal state x(T ) and/or T . 1. ∗ ∗ ∗ ∗ The optimal control problem is: find the flows x ≡ (x (t))t2[0;T ], u ≡ (u (t))t2[0;T ] such that u∗ maximizes the value functional Z T V [x; u] = F (u(t); x(t); t)dt + S(x(T );T ) 0 subject to the differential equation x_ = g(t; u(t); x=t)) given the initial value x(0) = x0 and possibly some additional information on the terminal state x(T ) and/or T . The two most important examples in economics are: • the consumption-asset accumulation problem Z T max u(C)t))e−ρtdt C 0 subject to W_ = Y − C(t) + rW where W (0) = W0 is the initial level of financial wealth and some other conditions can be imposed • the production investment problem for a firm Z T max PY (K(t)) − D(I(t);K(t))e−rtdt + S(K(T );T ) I 0 subject to K_ = I(t) − δK, where K(0) = K0 is the initial level of the capital stock. 1 ≡ dx(t) We use the common notation for time-derivatives x_(t) dt . Chapter 2 Calculus of variations The Calculus of Variations (CV) one important difference regarding static optimization. While statis optimization consists in finding an extremum number the calculus of variations consists in finding a best function. This function has its domain in a specified space, and the solution traces out a path in that space. Therefore, we can also say that the CV consists in finding a best path (in time, in some, or along any other domain). However, CV has an important analogy with static optimization. In static optimization we want to maximize a function (satisfying given constraints): it generally takes the form x∗ = arg maxf(x): The objective function, f(x) is a mapping f : number ! number We look for, in the range of the function f(:), the maximum, f(x∗), and find x∗ which is the solution to our problem (if it is unique). In CV we want to maximize a functional (satisfying given constraints): it takes the form Z T ∗ (x (t))t2T = arg maxV [x] = F (t; x(t); x_(t))dt: 0 3 Paulo Brito Mathematical Economics, 2018/19 4 The objective functional, V [x] is a mapping V : function ! number ∗ ∗ Again we look for, in the range of the functional V [x], the maximum V [x ], and find (x (t))t2T as the solution to our problem (if it is unique). Next we look at the CV problem from one of the two equivalent perspectives: • as a problem of finding the element x∗ 2 X on the space of functions, X , whose elements are mapping x : T ! X ⊆ R such that they maximize the functional V [x], i.e., V [x∗] = max V [x] x ∗ • as a problem of finding the path (x (t))t2T that maximizes the value functional Z T V [x∗] = max F (t; x(t); x_(t))dt: (x(t))t2T 0 Next we consider the following general calculus of variations problem with value functional Z T V [x] = F (t; x(t); x_(t))dt + S(x(T );T ) (2.1) 0 together with boundary and terminal time conditions. From now on we call x : T ! R the state variable, because it measures the system we are trying to optimize. Calculus of variations problems have the following components: 1. the value functional as in equation (2.1) has two components: the intertemporal ob- jective functional (in the independent variable is time) Z T J[x] = F (t; x(t); x_(t))dt 0 and the scrap function S(x(T );T ). While J[x] evaluates a path S(x(T );T ) evaluates the terminal value of the state. We assume throughout that both F (:) and S(:) are continuous, differentiable and smooth (F 2 C2(R3) and S 2 C2(R2)) in all its arguments and that F (:) concave in (x; x_); Paulo Brito Mathematical Economics, 2018/19 5 2. boundary conditions always involve a fixed initial state, x(0) = x0, and one of the following conditions regarding the terminal state and time: (a) fixed terminal state and time x(T ) = xT and T fixed (b) free terminal state and T fixed (c) strongly constrained terminal state (x(T );T ) = 0 and T fixed (d) weakly constrained terminal state (x(T );T ) ≥ 0 and T fixed (e) strongly constrained terminal state (x(T );T ) = 0 and T free (f) weakly constrained terminal state (x(T );T ) ≥ 0 and T free We can also consider other side constraints over the state of the system: 1. iso-perimetric constraints Z T G(t; x(t))dt ≤ 0 0 2. instantaneous constraints h(t; x(t); x_(t)) ≤ 0; for t 2 (0;T ) We will consider the following problems: • the simplest problem (see section 2.1.1): find maxx V [x] such that T is finite and fixed and x(0) = x0 and x(T ) = xT are given; • the simplest free terminal state problem: find maxx V [x] such that T is finite and given and x(0) = x0 is given and x(T ) is free; • infinite horizon discounted problem: F (x(t); x_(t); t) = e−ρtf(x(t); x_(t)), where ρ > 0 and T = 1 Paulo Brito Mathematical Economics, 2018/19 6 2.1 The simplest problems 2.1.1 Fixed end state and time ∗ ∗ The problem: find a path x = (x (t))t2T that solves the problem Z T V [x∗] = max F (t; x(t); x_(t))dt (x(t)) 8 t2[0;T ] 0 > >x(0) = x given t = 0 <> 0 (CV1) subject to >x(T ) = xT given t = T > > :T given We say that x is admissible if it verifies the restrictions of the problem, i.eif x 2 X where is the set of functions satisfying the initial and terminal conditions X = fx(t) 2 R : x(0) = ∗ x0; and x(T ) = xT ]. We say that a path x is optimal if it is admissible and it maximizes the functional (2.1); therefore it is a solution of the problem (CV1). Proposition 1 (First-order necessary conditions for problem (CV1)). Assume that x∗ is optimal. Then it verifies the Euler equation together with the boundary conditions 8 > ∗ ∗ >F (t) = d (F (t)) ; 0 ≤ t ≤ T (Euler-Lagrange equation) <> x dt x_ x∗(0) = x ; t = 0 (boundary condition: initial) (2.2) > 0 > :> ∗ x (T ) = xT ; t = T (boundary condition: terminal) ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ We use the notations F (t) = F (t; x (t); x_ (t)), Fx (t) = Fx(t; x (t); x_ (t)) and Fx_ (t) = Fx_ (t; x (t); x_ (t)) where ∗ ∗ @F (t; x(t); x_(t)) Fx(t; x (t); x_ (t)) ≡ @x x=x∗ and @F (t; x(t); x_(t)) Fx_ (t; x(t); x_(t)) ≡ : @x_ x=x∗ Paulo Brito Mathematical Economics, 2018/19 7 ∗ ∗ ∗ Denoting the second order partial derivatives, evaluated at the optimal path x , by Fxx, Fx_x_ ∗ and Fxx_ we observe that d F ∗ = F ∗ + F ∗ x_ ∗ + F ∗ x¨∗: dt x_ xt_ xx_ x_x_ This implies that the Euler equation can be written as a second order ordinary differential equation, ∗ ∗ 6 if Fx_x_ x¨ = 0 ∗ ∗ ∗ ∗ ∗ ∗ ≤ ≤ Fx = Fxt_ + Fxx_ x_ + Fx_x_ x¨ ; 0 t T: Example 1: the cake eating problem Let a cake have size W0 at the initial time t = 0 and assume we want to eat the cake completely until time T > 0.