Macroeconomics Lecture 5: dynamic programming methods, part three Chris Edmond 1st Semester 2019 1 This class Principle of optimality • – is solving the Bellman equation the same as solving the original sequence problem? Properties of the value function [skim this, for reference] • – using the maximum theorem and the contraction mapping theorem to deduce properties of the value function – what conditions are needed for value function to be increasing? concave? differentiable? 2 Sequence vs. recursive formulations General form of the sequence problem is to find • 1 t sup β F (xt,xt+1) (S) xt+1 t1=0 t=0 { } X subject to a sequence of constraint sets x Γ(x ),t=0, 1, 2,... t+1 2 t and given initial condition x X 0 2 Corresponding to this is the recursive problem • v(x)= sup F (x, y)+βv(y) for all x X (R) y Γ(x) 2 2 ⇥ ⇤ 3 Principle of optimality Do the solutions to these problems coincide? • General idea is that • (i) solution v to (R) evaluated at x = x0 gives the sup in (S), and (ii) asequence x 1 attains the sup in (S) if and only if { t+1}t=0 v(xt)=F (xt,xt+1)+βv(xt+1),t=0, 1, 2,... Bellman called this general idea the Principle of Optimality • What assumptions on the primitives deliver this result? • 4 Four primitives State space X Rn • ✓ Correspondence (set-valued function) Γ:X X describing • ◆ feasible choices as function of current state x X 2 The graph of this correspondence is A graph(Γ) = (x, y) X X : y Γ(x) ⌘ { 2 ⇥ 2 } Return function F : A R giving flow payoff • ! Discount factor β 0 • ≥ 5 Feasible plans Let ⇡(x ) denote set of feasible plans x starting from x • 0 { t+1}t1=0 0 ⇡(x )= x 1 : x Γ(x ),t=0, 1, 2,... 0 { t+1}t=0 t+1 2 t n o Let x ⇡(x ) denote a typical feasible plan • 2 0 6 Ensuring sequence problem is well-defined Assumption 1. Constraint set Γ(x) nonempty for all x X • 2 n t Assumption 2. The objective limn t=0 β F (xt,xt+1) exists • !1 for all x0 X and all x ⇡(x0) 2 2 P Remark. A sufficient condition for Assumption 2 is that F (x, y) is bounded and 0 <β<1. But weaker conditions often work too 7 Ensuring sequence problem is well-defined Now define partial sums of the form • n u (x) βtF (x ,x ), x ⇡(x ) n ⌘ t t+1 2 0 t=0 X By Assumption 2, we can also define the limit u(x) lim un(x) n ⌘ !1 This allows us to define a supremum function v by • ⇤ v⇤(x0) sup u(x) (S) ⌘ x ⇡(x0) 2 8 (S) (R) ) Under Assumptions 1 and 2, the v that solves (S) also solves (R) • ⇤ What about the converse? • Suppose v solves (R), does v also solve (S)? That is, does v = v ? • ⇤ 9 Partial converse In general no • But suppose v solves (R) and satisfies boundedness condition • n lim β v(xn)=0, for all x ⇡(x0) and x0 X (B) n !1 2 2 Then v = v⇤ In other words, v solves (R) but (R) may have other solutions • ⇤ If (R) has other solutions, they violate boundedness condition (B) • 10 Maximum theorem Let X Rn and Y Rm and let the function f : X Y R be ✓ ✓ ⇥ ! continuous and let the correspondence Γ:X ◆ Y be compact-valued and continuous. Then (i) the function h(x) max f(x, y) ⌘ y Γ(x) 2 is continuous and (ii) the solution correspondence G : X ◆ Y given by G(x) argmax f(x, y)= y Γ(x):f(x, y)=h(x) ⌘ y Γ(x) { 2 } 2 is nonempty, compact-valued and upper hemicontinuous (see appendix) Remark. If f(x, y) is strictly concave in y and Γ(x) is convex, the solution correspondence is single-valued and we write it g(x). Since g(x) is single valued and upper hemicontinuous, it is continuous 11 Bounded returns Let’s now focus on the functional equation • v(x) = max F (x, y)+βv(y)], for all x X y Γ(x) 2 2 ⇥ with some additional structure Assumption 3. X Rn is convex and the correspondence • ✓ Γ:X ◆ X is nonempty, compact-valued and continuous Assumption 4. F : A R is bounded, continuous and 0 <β<1 • ! 12 Bounded returns Recall C(X) is set of bounded continuous functions f : X R • ! We can now conclude that the operator Tf(x) = max F (x, y)+βf(y)], for all x X y Γ(x) 2 2 ⇥ maps C(X) to C(X). That is, if f C(X) then Tf C(X) too 2 2 This is because, under Assumptions 3 and 4: • – the RHS maximizes a continuous function over a compact-valued, continuous correspondence, hence (i) the maximum is attained, and (ii), by the maximum theorem, Tf is continuous – both F and f are bounded, so Tf is bounded 13 Bounded returns So T maps C(X) to C(X) • Moreover T satisfies Blackwell’s conditions, so T is a contraction • Since C(X) is a complete metric space and T is a contraction, • there is a unique fixed point v = Tv C(X) that solves (R) 2 Since Assumptions 1, 2 and the boundedness condition (B) are • satisfied, this v also solves (S) What else can we say about v? • 14 A useful result Let (S, d) be a complete metric space and let T : S S be a • contraction mapping with fixed point v S. Then ! 2 (i) if S0 is a closed subset of S and T (S0) S0 then v S0,and ✓ 2 (ii) if T (S0) S00 S,thenv S00 ✓ ✓ 2 15 Examples Let • – S denote the set of continuous functions – S0 denote the set of nondecreasing functions, and – S00 denote the set of strictly increasing functions Then • – if T maps f S to Tf S0 then v S0 so v nondecreasing 2 2 2 – if T maps f S0 to Tf S00 then v S00 so v strictly increasing 2 2 2 16 General idea Suppose f has some property and T preserves this property. • P Then Tf also has P Then by induction T nf also has has for any finite n =0, 1,... • P Then if is a property that is preserved by uniform convergence, • P the fixed point v = Tv also has P 17 Monotonicity Suppose we also have the following monotonicity assumptions • Assumption 5. F (x, y) strictly increasing in x for each y • Assumption 6. Γ(x) increasing in that if x x then Γ(x) Γ(x ) • 0 ✓ 0 18 Value function v is strictly increasing Suppose the primitives X, Γ,F,β satisfy Assumptions 3, 4, 5 and 6 • and suppose v is the unique solution to v(x) = max F (x, y)+βv(y)], for all x X y Γ(x) 2 2 ⇥ Then the value function v is strictly increasing • 19 Proof (sketch) Let f C(X) and let x<x. Then we have • 2 0 Tf(x) = max F (x, y)+βf(y) y Γ(x) 2 ⇥ ⇤ max F (x, y)+βf(y) y Γ(x ) 2 0 ⇥ ⇤ < max F (x0,y)+βf(y) = Tf(x0) y Γ(x ) 2 0 ⇥ ⇤ Hence T maps bounded continuous functions f into bounded • continuous and strictly increasing functions Tf 20 Proof (sketch) Now let C (X) denote the set of bounded continuous • 0 nondecreasing functions on X and let C (X) C (X) denote the 00 ⇢ 0 set of bounded continuous strictly increasing functions Then since T maps f C (X) to Tf C (X) the fixed point • 2 0 2 00 v = Tv C (X) too. Hence v is strictly increasing 2 00 21 Curvature Assumption 7. F is strictly concave, that is, for any ✓ (0, 1) • 2 F ( ✓(x, y)+(1 ✓)(x0,y0))>✓F(x, y)+(1 ✓)F (x0,y0) − − for any (x, y) and (x0,y0) in the graph of Γ Assumption 8. Γ is convex in the sense that for any ✓ (0, 1) • 2 y Γ(x) and y0 Γ(x0) 2 2 implies ✓y +(1 ✓)y0 Γ( ✓x +(1 ✓)x0 ) − 2 − for any x, x X 0 2 (since X is convex, equivalent to assuming graph of Γ is convex) 22 Value function v is strictly concave Suppose the primitives X, Γ,F,β satisfy Assumptions 3, 4 and 7, 8 • and suppose v is the unique solution to v(x) = max F (x, y)+βv(y)], for all x X y Γ(x) 2 2 ⇥ Then v is strictly concave • Moreover there is a continuous single-valued policy function • y = g(x) that attains the maximum 23 Proof (sketch) Let x = x and let x = ✓x +(1 ✓)x for ✓ (0, 1) • 6 0 ✓ − 0 2 Let y Γ(x) be such that Tf(x)=F (x, y)+βf(x) and likewise • 2 Let y Γ(x ) be such that Tf(x )=F (x ,y)+βf(x ) 0 2 0 0 0 0 0 Then • Tf(x ) F (x ,y )+βf(y ) ✓ ≥ ✓ ✓ ✓ >✓ F (x, y)+βf(y) +(1 ✓) F (x0,y0)+βf(y0) − ⇥ ⇤ ⇥ ⇤ = ✓Tf(x)+(1 ✓)Tf(x0) − Hence Tf is strictly concave 24 Proof (sketch) Now let C (X) denote the set of bounded continuous concave • 0 functions on X and let C (X) C (X) denote the set of bounded 00 ⇢ 0 continuous strictly concave functions Then since T maps f C (X) to Tf C (X) the fixed point • 2 0 2 00 v = Tv C (X) too.