Discrete Choice Models Models Discrete-Choice Dynamic-Programming Models References

Estimating Dynamic Programming Models

Katsumi Shimotsu1 Ken Yamada2

1Department of Economics, Hitotsubashi University

2School of Economics, Singapore Management University

Japanese Economic Association Spring Meeting Tutorial Session (May, 2011) Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

Introduction to Part II

• The second part focuses on econometric implementation of discrete choice dynamic programming models. • A simple machine replacement model is estimated using the nested fixed point algorithm (Rust, 1987). • For more applications, see Aguirregabiria and Mira (2010) and Keane, Todd, and Wolpin (2010).

• For audience with different backgrounds, I will briefly review 1. Discrete choice models and 2. Numerical methods for (i) maximum likelihood estimation and (ii) dynamic programming. Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

Outline

Discrete Choice Models The Random Utility Model Maximum Likelihood Estimation

Dynamic Programming Models Machine Replacement Numerical Dynamic Programming

Discrete-Choice Dynamic-Programming Models The Nested Fixed Point Algorithm The Nested Pseudo Likelihood Algorithm

References Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

Outline

Discrete Choice Models The Random Utility Model Maximum Likelihood Estimation

Dynamic Programming Models Machine Replacement Numerical Dynamic Programming

Discrete-Choice Dynamic-Programming Models The Nested Fixed Point Algorithm The Nested Pseudo Likelihood Algorithm

References Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

The Random Utility Model • Consider a static problem in which the agent chooses among J alternatives, such as transportation mode and brand choice. • Assume that the choice-specific utility can be expressed as

Vij = uij + εij i = 1,...,N, j = 0...,J,

where uij is a deterministic component while εij is a stochastic component.

• A simple example is uij = xij θ, where xij is a vector of observed characteristics such as price and income.

• Let A = {1,...,J} denote the choice set. The choice probability is Z ( ) P (a) = 1 a = argmax (uij + εij ) f (εi )dεi , ε j∈A 0 where f (εi ) is a joint density of εi = (εi1,...,εiJ ) . Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

The Conditional Logit Model

• Assume that each εj is independently, identically distributed −ε −ε extreme value. The density is f (εj ) = e j exp(−e j ). The choice probability follows the logit model (McFadden, 1981).

exp(uij ) exp(uij − ui1) pij = P (a = j) = J = J ∑j=1 exp(uij ) 1 + ∑j=2 exp(uij − ui1) • Note that this is a system of J − 1 equations. For J = 2,

exp(ui2 − ui1) pi2 = = Λ(ui2 − ui1). 1 + exp(ui2 − ui1) There exists a mapping between the utility differences and choice probabilities.   −1 pi2 ui2 − ui1 = Λ (pi2) or ui2 − ui1 = ln 1 − pi2 See Hotz and Miller (1993) for the details on invertibility. Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

Outline

Discrete Choice Models The Random Utility Model Maximum Likelihood Estimation

Dynamic Programming Models Machine Replacement Numerical Dynamic Programming

Discrete-Choice Dynamic-Programming Models The Nested Fixed Point Algorithm The Nested Pseudo Likelihood Algorithm

References Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

MLE

• The choice probability is

exp(uij ) P (a = j) = J ∑j=1 exp(uij )

• Suppose Vij = xij θ + εij . The log-likelihood function is

N N J l (θ) = ∑ li (θ) = ∑ ∑ 1(ai = j)lnP (ai = j|xij ). i=1 i=1 j=1

• If the model is correctly specified,

 2 −1! d ∂ l θb → N θ,E − . ∂θ∂θ 0 Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

Numerical Maximization (Train, 2009) Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

Numerical Maximization (Train, 2009) Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

Numerical Methods for MLE

• The Newton-Raphson procedure uses the following formula.

−1 θk+1 = θk + −Hk gk ,

2  0 where gk = ∂l/∂θ| and Hk = ∂ l ∂θ∂θ . θk θk

• Quasi-Newton methods: [fval,theta,g,H] = csminwel(0fun0,theta0,H0,g,crit,nit) by Christopher Sims

• Non-gradient based algorithm (simplex methods): [theta,fval] = fminsearch(0fun0,theta0,options) Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

Extension to Dynamic Models • We may be concerned with dynamics in some applications such as firm’s adjustment of labor and capital, educational and occupational choices. • The value function V is the unique solution to the :

V (xt ,εt ) = max {Vjt = V (xt ,εt ,at = j)}, j∈A where the choice-specific value function (including ε) is

V (xt ,εt ,at ) = u (xt ,at ) + εt (at ) + βE[V (xt+1,εt+1)|xt ,εt ,at ] = {u (xt ,at ) + βE[V (xt+1,εt+1)|xt ,εt ,at ]} + εt (at ) = v (xt ,at ) + εt (at ) for β ∈ (0,1). The model considered here is a natural extension of the conditional logit model (or the random utility model) to a dynamic setting. Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

Outline

Discrete Choice Models The Random Utility Model Maximum Likelihood Estimation

Dynamic Programming Models Machine Replacement Numerical Dynamic Programming

Discrete-Choice Dynamic-Programming Models The Nested Fixed Point Algorithm The Nested Pseudo Likelihood Algorithm

References Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

Plant Investment Decisions • Here we consider the plant’s investment decision problem set out by Kasahara at UBC. The tradeoff is that investment (machine replacement), a ∈ {0,1}, 1. requires large fixed costs for now but 2. increases profits and lowers replacement costs in the future. • The profit function is

u (zt ,at−1,at ) = R (zt ,at ) − FC (at ,at−1),

where R is revenues net of variable input costs, zt is plant’s productivity, and FC is fixed costs. 2 • Assume that zt = ρzt−1 + ηt , where ηt ∼ N 0,ση . Specify the profit function as

u (zt ,at−1,at ) = u (zt−1,at−1,at )

= exp(θ0 + θ1zt−1 + θ2at ) − [θ3at + θ4 (1 − at−1)at ]. Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

Transition Probability

• Let Z = {z1,...,zg ,...,zG }. • Denote the transition probability by

qgh = q (zt = zh|zt−1 = zg ) = Pr(z ∈ [zh,zh+1]|z ∈ [zg ,zg+1]).

By the Tauchen method, this can be approximated by   Φ((z1 − ρzg )/ση ) if h = 1, qgh = Φ((zh − ρxg )/ση ) − Φ((zh−1 − ρzg )/ση ) if 1 < h < G,  1 − Φ((zG − ρzg )/ση ) if h = G,

where zg = (zg + zg+1)/2 and Φ is the standard normal cdf. Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

The Choice-Specific Value Function

• Define the integrated value function as Z V (zt−1,at−1) = V (zt−1,at−1,εt )dF (εt ). εt

• Assume that each εt (at ) is iid extreme value.

J ! V (zt−1,at−1) = γ + ln ∑ exp(v (zt−1,at−1,at = j)) , j=1

where γ is the Euler constant. • The choice-specific value function is

v (zt−1,at−1,at ) = u (zt−1,at−1,at )+β ∑q (zt |zt−1)V (zt−1,at−1). xt Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

Outline

Discrete Choice Models The Random Utility Model Maximum Likelihood Estimation

Dynamic Programming Models Machine Replacement Numerical Dynamic Programming

Discrete-Choice Dynamic-Programming Models The Nested Fixed Point Algorithm The Nested Pseudo Likelihood Algorithm

References Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

Infinite Horizon Problem • In the infinite-horizon problem with a finite number of states, the Bellman equation is a system of nonlinear equations.

" G # Vg = max u (xg ,a) + β qgh (a)Vh , a∈A ∑ h=1 where a ∈ {1,...,J} is a choice variable, x ∈ {x1,...,xg ,,...,xG } is a state variable, and qgh (a) is the transition probability from state g to state h. 0 • Let a = (a1,...,aG ) denote the policy function and a 0 u = (u (x1,a1),...,u (xG ,aG )) the return. Then, in vector notation, Va = ua + βQaVa, which leads to the solution: Va = (I − βQa)−1 ua, a where the ghth element of Q is qgh (a). For details, see Judd (1998) and Adda and Cooper (2003). Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

Value Iteration Algorithm

1. Choose a grid X = {x1,...,xg ,,...,xG }, where xg < xh for g < h, and specify u (x,a) and qgh (a). 2. Make an initial guess V 0 and choose stopping criterion c > 0. 3. For g = 1,...,G, compute

( G ) k+1 k Vg = max u (xg ,a) + β qgh (a)Vh . a∈A ∑ h=1

4. If V k − V k+1 < c, stop; else go to step 3. Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

Policy Iteration Algorithm

1. Choose a grid X = {x1,...,xg ,,...,xG }, where xg < xh for g < h, and specify u (x,a) and qgh (a). 2. Make an initial guess V 0 and choose stopping criterion c > 0. 0 k+1  k+1 k+1 3. Compute a = a1 ,...,aG , where

( G ) k+1 k ag = argmax u (xg ,a) + β qgh (a)Vh . a∈A ∑ h=1

0 k+1   k+1  k+1 4. Compute u = u x1,a1 ,...,u xG ,aG . 5. Compute  k+1 −1 Vk+1 = I − βQa uk+1.

6. If V k − V k+1 < c, stop; else go to step 3. Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

Outline

Discrete Choice Models The Random Utility Model Maximum Likelihood Estimation

Dynamic Programming Models Machine Replacement Numerical Dynamic Programming

Discrete-Choice Dynamic-Programming Models The Nested Fixed Point Algorithm The Nested Pseudo Likelihood Algorithm

References Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

The Nested Fixed Point Algorithm (Rust, 1987)

• The conditional choice probability takes the form:

exp(u (x,a,θ) + β V (x 0,ε0)) P (a|x) = E . J 0 0 ∑j=1 exp(u (x,j,θ) + βEV (x ,ε )) • The transition probability can be obtained separately.

1. Inner loop: Given θ, compute the value function V using the value iteration algorithm or the policy iteration algorithm, and obtain the conditional choice probability P. 2. Outer loop: Given P, obtain a pseudo-likelihood estimate of θ using the Newton method. 3. Repeat steps 1 and 2 until convergence. Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

Outline

Discrete Choice Models The Random Utility Model Maximum Likelihood Estimation

Dynamic Programming Models Machine Replacement Numerical Dynamic Programming

Discrete-Choice Dynamic-Programming Models The Nested Fixed Point Algorithm The Nested Pseudo Likelihood Algorithm

References Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

Hotz and Miller (1993) and Aguirregabiria and Mira (2002) o • Denote the optimal decision rule by djt = 1{at = j|xt ,εt }.

( T J ) τ−t o V (xt ) = E ∑ ∑ β djt [u (xt ,at ) + εt (at )] τ=t j=1 The Bellman equation can be rewritten as " # 0  0 V (x) = ∑P (a|x) u (x,a) + E[ε|x,a] + β ∑q x x,a V x . a x 0

• The system of equations can be solved for the value function. In matrix notation,  −1 V = I − β ∑P (a) ◦ Q (a) ∑P (a)◦[u (a) + (γ − lnP (a))], a a where ◦ is the element-by-element product. Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

The Conditional Logit Form • Suppose u (x,a) = z (x,a)α. The policy iteration operator is

exp(ez (x,a)α +e(x,a)) Ψ(P) = J , ∑j=1 exp(ez (x,a)α +e(x,a)) where

0  0  ez (x,a) = z (x,a) + β ∑q x x,a Wz x ,P , x 0 0  0  e(x,a) = β ∑q x x,a We x ,P , x 0 " #−1 0  Wz x ,P = I − β ∑P ◦ Q ∑P ◦ z, j a  −1 0  We x ,P = I − β ∑P ◦ Q ∑P ◦ (γ − ln(P)). a a Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

The Nested Pseudo Likelihood Algorithm

• Make an initial guess P0 and choose the number of iterations K.

0 0 k−1 1. Given ez (x,a) = z (x,a) + β ∑x 0 q (x |x,a)Wz x ,P and 0 0 k−1 e(x,a) = β ∑x 0 q (x |x,a)We x ,P , obtain a pseudo-likelihood estimate of αk as

N   ak = argmax Ψ Pk−1 . α ∑ i=1

2. Given ak , update P using   Pk = Ψ Pk−1 .

3. Iterate steps 1 and 2 until k = K. Discrete Choice Models Dynamic Programming Models Discrete-Choice Dynamic-Programming Models References

References

1. Adda, J. and R. Cooper, (2003): Dynamic Economics: Quantitative Methods and Applications, MIT Press. 2. Aguirregabiria, V. and P. Mira, (2002): “Swapping the nested fixed point algorithm: A class of estimators for discrete Markov decision models,” Econometrica. 3. Aguirregabiria, V. and P. Mira, (2010): “Dynamic discrete choice structural models: A survey,” Journal of Econometrics. 4. Hotz, J. and R. A. Miller, (1993). “Conditional choice probabilities and the estimation of dynamic models,” Review of Economic Studies. 5. Judd, K. L. (1998): Numerical Methods in Economics, MIT Press. 6. Keane, M. P., P. E. Todd, and K. I. Wolpin, (2010). “The structural estimation of behavioral models: discrete choice dynamic programming methods and applications,” Handbook of Labor Economics, Vol. 4A. 7. McFadden, D. L. (1981): “Structural discrete probability models derived from theories of choice,” in C. F. Manski and D. L. McFadden (eds.) Structural Analysis of Discrete Data and Economic Applications, MIT Press. 8. Rust, J. (1987): “Optimal replacement of GMC bus engines: An empirical model of Harold Zurcher” Econometrica. 9. Train, K. (2009): Discrete Choice Methods with Simulation, Cambridge University Press.