Economics 2010c: Lecture 8 Brownian Motion and Continuous Time Dynamic Programming
David Laibson
9/25/2014 Outline: Continuous Time Dynamic Programming
1. Continuous time random walks: Wiener Process
2. Ito’s Lemma
3. Continuous time Bellman Equation 1 Brownian Motion
Consider a continuous time world [0 ) ∈ ∞ Imagine that every ∆ intervals, a process () either goes up or down: + with prob ∆ ( + ∆) ()= ≡ − with prob 1 ( − ≡ − )
Here ∆ is a fixed interval of time. Later we will let ∆ 0 → (∆)= + ( )=( ) − − [(∆)2]=2 + 2 = 2
(∆)=[∆ ∆]2 = [(∆)2] [∆]2 =42 − − Time span of length implies = ∆ steps in ∆ So () (0) is • − a binomial random variable.
For example, suppose =5and = = 1 • 2 5 Pr [() (0) = 5]= 05 =003 − − 0 ³5´ Pr [() (0) = 3]= 14 =016 − − 1 ³5´ Pr [() (0) = 1]= 23 =031 − − 2 ³5´ Pr [() (0) = +1]= 32 =031 − 3 ³5´ Pr [() (0) = +3]= 41 =016 − 4 ³5´ Pr [() (0) = +5]= 50 =003 − 5 ³ ´ Generally, the probability that • () (0) = ()()+( )( ) − − − is − ³ ´
() (0) is a binomial random variable with • − [() (0)] = ( ) = ( )∆ − − − [() (0)] = 42 = 42∆ − We will now vary ∆ Begin by letting = √∆ 1 = 1+ √∆ 2 ∙ ¸ The following five implications follow: 1 =1 = 1 √∆ − 2 ∙ − ¸ ( )= √∆ − [() (0)] = √∆√∆∆ = − 1 2 2∆ [() (0)] = 4 1 ∆ 2 (as ∆ 0) − µ4¶ à − µ¶ ! ∆ −→ → Brownian Motion (∆t = 1, σ = 1, α = 0) 10
8
6
4
2
0 x(t)
-2
-4
-6
-8
-10 0 10 20 30 40 50 60 70 80 90 100 Time (t) Brownian Motion (∆t = .1, σ = 1, α = 0) 10
8
6
4
2
0 x(t)
-2
-4
-6
-8
-10 0 10 20 30 40 50 60 70 80 90 100 Time (t) Brownian Motion (∆t = .01, σ = 1, α = 0) 10
8
6
4
2
0 x(t)
-2
-4
-6
-8
-10 0 10 20 30 40 50 60 70 80 90 100 Time (t) Brownian Motion (∆t = .00001, σ = 1, α = 0) 10
8
6
4
2
0 x(t)
-2
-4
-6
-8
-10 0 10 20 30 40 50 60 70 80 90 100 Time (t) 1. Vertical movements proportional to √∆ (not ∆)
2. [() (0)] ( 2) since Binomial Normal − → →
3. “Length of curve during 1 time period” = 1 √∆ = 1 ∆ √∆ →∞
4. ∆ = √∆ = . Time derivative, ,doesn’texist. ∆ ± ∆ √±∆ → ±∞ ³ ´
(∆) 22 5. ∆ = ∆ = .Sowewrite ()=.
4 1 1 ()2∆ 2∆ 6. (∆) = 4 − 2.Sowewrite ()=2. ∆ ³ ´³ ∆ ´ → When we let ∆ converge to zero, the limiting process is called a continuous time random walk with (instantaneous) drift and (instantaneous) variance 2 We generated this continuous-time stochastic process by building it up as a limit case. We could have also just defined the process directly.
Definition 1.1 If a continuous time stochastic process, () is a Wiener Process,then( ) () satisfies the following conditions: 0 − 1. ( ) () (0 ) 0 − ∼ 0 − 2. If , ≤ 0 ≤ 00 ≤ 000 (( ) ())(( ) ( )) =0 0 − 000 − 00 h i You can also think of the two condition as: 1. ( ) ()=√ where (0 1) 0 − 0 − ∼ 2. non-overlapping increments of are independent Summary:
A Wiener process is a continuous time random walk with zero drift and • unit variance.
() has the Markov property: the current value of the process is a suffi- • cient statistic for the distribution of future values.
() ((0)) sothevarianceof() rises linearly with • ∼ Generalization: Let () be a Wiener Process. Let () be another continuous time stochastic process such that, ∆ lim = ( ) i.e. ()=( ) ∆ 0 ∆ → ∆ lim = ( )2 i.e. ()=( )2 ∆ 0 ∆ → We summarize these properties by writing: = ( ) + ( ) This is called an Ito Process. Important examples:
= + (random walk with drift and variance 2) •
= + (geometric random walk with proportional drift • and proportional variance 2) 2 Ito’s Lemma
Our goal: work with functions that take an Ito Process as an argument.
Suppose that the price of oil follows an Ito Process: • = ( ) + ( )
The value of an oil well will depend on the price of oil and time: ( ) •
We would like to be able to write the stochastic process that describes the • evolution of : =ˆ( ) + ˆ( ) whichwewillcallthetotaldifferential of Theorem 2.1 (Ito’s Lemma) Let () be a Wiener Process. Let () be an Ito Process with = ( ) + ( ) Let = ( ) then 12 = + + ( )2 2 2 12 = + ( )+ ( )2 + ( ) " 2 2 # Proof: Using a Taylor expansion: 12 12 2 = + ()2 + + ()2 + + 2 2 2 2 Any deterministic term of order ()32 or higher is small relative to terms of order Any stochastic term of order or higher is small relative to terms of order √ So, ()2 =
= ( )()2 + ( ) =
()2 = ( )2()2 + = ( )2 + Combining these results, we have our key result: 12 = + + ( )2 2 2 ¥ 2.1 Intuition:
Assume ( )=0 (no drift in the Ito Process (): ()=0). •
Assume that () =0 (holding fixed, doesn’t depend on ). •
2 But, ( )=1 ( )2 =0 • 2 2 6
If is concave (convex), is expected to fall (rise) due to variation in •
For Ito Processes, ()2 behaves like ( )2 so the effect of concavity • (convexity) is of order and can not be ignored when calculating the differential of ()=log(),so = 1 and = 1 • 0 00 −2
= + • 12 = + ( )+ ( )2 + ( ) " 2 2 # 1 1 1 1 = 0+ + 22 + 2 ∙ 2 µ− ¶ ¸ 1 = 2 + ∙ − 2 ¸
falls below due to concavity of • 3 Continuous time Bellman Equation
Let ( )=instantaneous payoff function, where is state variable, is control variable and is time. Let 0 = + ∆ and 0 = + ∆ So ( )=max ( )∆ +(1+∆) 1 ( ) − 0 0 n o (1 + ∆) ( )=max (1 + ∆)( )∆ + ( ) 0 0 n o ( )∆ =max (1 + ∆)( )∆ + ( ) ( ) 0 0 − n o Multiply out and let ∆ 0 Terms of order ()2 =0 → ( ) =max ( ) + ( ) (*) { } Now substitute in for ( ) using Ito’s Lemma: 12 = + + 2 + " 2 2 # where = ( ), = ( ) and = ( ) + ( ) Since, =0 we have, 12 ( )= + + 2 " 2 2 # Substituting this expression into equation (*), we get 12 ( ) =max ( ) + + + 2 ( " 2 2 # ) which is a partial differential equation (in and ). Outline: Continuous Time Dynamic Programming
1. Continuous time random walks: Wiener Process
2. Ito’s Lemma
3. Continuous time Bellman Equation