Approximation Methods

Home , Approximation theory, Function approximation

Lecture Notes 3

In this chapter, we deal with a very important problem that we will encounter in a wide variety of economic problems: approximation of functions. Such a problem commonly occurs when it is too costly either in terms of time or complexity to compute the true function or when this function is unknown and we just need to have a rough idea of its main properties. Usually the only thing that is required then is to be able to compute this function at one or a few points and formulate a guess for all other values. This leaves us with some choice concerning either the local or global character of the approximation and the level of accuracy we want to achieve. As we will see in diﬀerent applications, choosing the method is often a matter of eﬃciency and ease of computing. Following Judd [1998], we will consider 3 types of approximation methods

1. Local approximation, which essentially exploits information on the value of the function in one point and its derivatives at the same point. The idea is then to obtain a (hopefully) good approximation of the function in a neighborhood of the benchmark point.

2. Lp approximations, which actually ﬁnd a nice function that is close to the function we want to evaluate in the sense of a Lp norm. Ideally, we would need information on the whole function to ﬁnd a good approximation, which is usually infeasible – or which would make the problem

1 of approximation totally irrelevant! Therefore, we usually rely on interpolation, which then appears as the other side of the same problem, but only requires to know the function at some points.

3. Regressions, which may be viewed as an intermediate situation between the two preceding cases, as it usually relies — exactly as in econometrics, on m moments to ﬁnd n parameters of the approximating function.

3.1 Local approximations

The problem of the local approximation of a function f : R R is to make −→ use of information about the function at a particular point x0 R, to produce ∈ a good approximation of f in a neighborhood of x0. Among the various available method 2 are of particular interest: the Taylor series expansion and Pad´e approximation.

3.1.1 Taylor series expansion

Taylor series expansion is certainly the most wellknown and natural approximation to any student.

The basic framework: This approximation relies on the standard Taylor’s theorem:

Theorem 1 Suppose F : R R is a Ck+1 function, then for x? Rn, we −→ ∈ have n ∂F F (x) = F (x?) + (x?)(x x?) ∂x i − i i=1 i n Xn 1 ∂F ? ? ? + (x )(xi1 xi1 )(xi2 xi2 ) + . . . 2 ∂xi1 ∂xi2 − − i=1 i1=1 Xn X n 1 ∂F ? ? ? + . . . (x )(xi1 xi1 ) . . . (xik xik ) k! ∂xi1 . . . ∂xi1 − − iX1=1 iXk=1 +O x x? k+1 k − k ³ ´ 2 The idea of Taylor expansion approximation is then to form a polynomial approximation of the function f as described by the Taylor’s theorem. This approximation method therefore applies to situations where the function is at least n times differentiable to get a n–th order approximation. If this is the case, then we are sure that the error will be at most of order O x x? k+1 .1. k − k In fact, we may look at Taylor series expansion from a sligh¡ tly differen¢ t perspective and acknowledge that it amounts to approximate the function by an infinite series. For instance in the one dimensional case, this amounts to write n F (x) α (x x?)k ' k − Xk=0 where α = 1 ∂F (x?). As n tends toward infinity, the latter equation may be k k! ∂xi understood as a power series expansion of the F in the neighborhood of x?. This is a natural way to think of this type of approximation if we just think of the exponential function for instance, and the way a computer delivers exp(x). Indeed, the formal definition of the exponential function is traditionally given by ∞ xk exp(x) ≡ k! Xi=0 The advantage of this representation is that we are now in position to give a very important theorem concerning the relevance of such approximation. Nevertheless, we need to report some preliminary definitions.

Deﬁnition 1 We call the radius of convergence of the complex power series, the quantity r deﬁned by

∞ k r = sup x : αkx < (| | ¯ ¯ ∞) ¯k=0 ¯ ¯X ¯ r therefore provides the maximal radius¯ of x ¯C for which the complex series ¯ ∈¯ converges. That is for any x C such that x < r, the series converges while ∈ | | it diverges for any x C such that x > r. ∈ | | 1 n −→ k O ` kf(x)k ∞ Let us recall at this point that a function f : R R is x if limx→0 kxk` < ¡ ¢ 3 Definition 2 A function F : Ω C C is said to be analytic, if for every ⊂ −→ x? Ω there exists a sequence α and a radius r such that ∈ k n F (x) = α (x x?)k for x x? < r k − k − k Xk=0 Definition 3 Let F : Ω C C be a function, and x? Ω. x? is a ⊂ −→ ∈ singularity of F is F is analytic on Ω x? but not on Ω. − { } For example, let us consider the tangent function tan(x). tan(x) is defined by the ratio of two analytic functions sin(x) tan(x) = cos(x) since cos(x) and sin(x) may be written as

∞ x2n cos(x) = ( 1)n − (2n)! i=0 X∞ x2n+1 sin(x) = ( 1)n − (2n + 1)! Xi=0 However, the function obviously admits a singularity at x? = π/2, for which cos(x?) = 0. Given these deﬁnition we can now enounce the following theorem:

Theorem 2 Let F be an analytic function in x C. If F or any derivative ∈ of F exhibits a singularity at xo C, then the radius of convergence in the ∈ complex plane of the Taylor series expansion of F in the neighborhood of x?

∞ 1 ∂kF (x?)(x x?)k k! ∂xk − Xk=0 is bounded from above by x? xo . k − k This theorem is extremely important as it gives us a guideline to use Taylor series expansions, as it actually tells us that the series at x? cannot deliver any accurate, and therefore reliable, approximation for F an any point farther

4 away from x? than any singular point of F . An extremely simple example to understand this point is provided by the approximation of the function F (x) = log(1 x), where x ( , 1), in a neighborhood of x? = 0. First of − ∈ −∞ all, note that xo = 1 is a singular value for F (x). The Taylor series expansion of F (x) in the neighborhood of x? = 0 writes

∞ xk log(1 x) ϕ(x) − ' ≡ − k Xk=1 What the theorem tells us is that this approximation may be used for values of x such that x x? is below x? xo = 0 1 = 1, that is for x such k − k k − k k − k that 1 < x < 1. In other words, the radius of convergence of the Taylor − approximation to this function is r = 1. As an numerical illustration, we report in table 3.1, the “true” value of log(1 x), its approximate value using − 100 terms in the summation, and the absolute deviation of this approximation from the “true” value.2 As can be seen from the table, as soon as x? xo k − k approaches the radius, the accuracy of the approximation by the Taylor series expansion performs more and more poorly.

Table 3.1: Taylor series expansion for log(1 x) − x log(1 x) ϕ (x) ε − 100 -0.9999 0.69309718 0.68817193 0.00492525 -0.9900 0.68813464 0.68632282 0.00181182 -0.9000 0.64185389 0.64185376 1.25155e-007 -0.5000 0.40546511 0.40546511 1.11022e-016 0.0000 0.00000000 0.00000000 0 0.5000 -0.69314718 -0.69314718 2.22045e-016 0.9000 -2.30258509 -2.30258291 2.18735e-006 0.9900 -4.60517019 -4.38945277 0.215717 0.9999 -9.21034037 -5.17740221 4.03294

2The word true lies between quotes as it has been computed by the computer and is therefore an approximation!

5 The usefulness of the approach: This approach to approximation is particularly useful and quite widespread in economic dynamics. For instance, anyone that has ever tried to study the dynamic properties of any macro model — let’s say the optimal growth model — has once encountered the method. Let’s take for instance the optimal growth model which dynamics may be summarized — when preferences are logarithmic and technology is Cobb–Douglas — by the two following equations3

k = kα c + (1 δ)k (3.1) t+1 t − t − t 1 1 α−1 = β αkt+1 + 1 δ (3.2) ct ct+1 − ¡ ¢ It is a widespread practice to linearize of log–linearize such an economy around the steady state, which we will deﬁne later on, to study the local dynamic properties of the equilibrium. Let’s assume that the steady state has already ? ? been found and is given by ct+1 = ct = c and kt+1 = kt = k for all t.

Linearization: let us denote by kt the deviation of the capital stock kt with respect to its steady state level in period t, such that k = k k?. Likewise, t t − ? b we define ct = ct c . The first step of linearization is to reexpress the system − b in terms of functions. Here, we can define b k kα + c (1 δ)k t+1 − t t − − t F (k , c , k , c ) = t+1 t+1 t t  1 β 1 αkα−1 + 1 δ  ct − ct+1 t+1 − ¡ ¢ and then build the Taylor expansion

F (k , c , k , c ) F (k?, c?, k?, c?) + F (k?, c?, k?, c?)k + F (k?, c?, k?, c?)c t+1 t+1 t t ' 1 t+1 2 t+1 ? ? ? ? ? ? ? ? +F3(k , c , k , c )kt + F4(k , c , k , c )ct b b 3 Since we just want to make the case of Taylorb series expansions, we dob not need to be any more precise on the origin of these two equations. Therefore, we will take them as given, but we will come back to their determination in the sequel.

6 We therefore just have to compute the derivatives of each sub–function, and realize that the steady state corresponds to a situation where F (k?, c?, k?, c?) = 0. This yields the following system k αk?α−1k + c (1 δ)k = 0 t+1 − t t − − t 1 c + β 1 αk?α−1 + 1 δ c β 1 α(α 1)k?α−2k = 0 ( − c?2 t c?2 − t+1 − c? − t+1 b b b b which simpliﬁes to ¡ ¢ b b b ?α−1 kt+1 αk kt + ct (1 δ)kt = 0 − − − ? c + β αk?α−1 + 1 δ c βα(α 1) c k?α−1k = 0 ( − t − t+1 − − k? t+1 b b b b We then have to solv¡ e the implied¢linear dynamic system, bu this is another b b b story that we will deal with in a couple of chapters.

Log–linearization: Another common practice is to take a log–linear approximation to the equilibrium. Such an approximation is usually taken because it delivers a natural interpretation of the coeﬃcients in front of the variables: these can be interpreted as elasticities. Indeed, let’s consider the following one–dimensional function f(x) and let’s assume that we want to take a log–linear approximation of f around x?. This would amount to have, as deviation, a log–deviation rather than a simple deviation, such that we can deﬁne x = log(x) log(x?) − Then, a restatement of the problem is in order, as we are to take an approxi- b mation with respect to log(x):

f(x) = f(exp(log(x))) which leads to the following ﬁrst order Taylor expansion

f(x) f(x?) + f 0(exp(log(x?))) exp(log(x?))x = f(x?) + f 0(x?)x?x ' If we apply this technic to the growth model, we end up with the following b b system k 1−β(1−δ) k + 1−β(1−δ) δ c (1 δ)k = 0 t+1 − β t αβ − t − − t ( ct + ct+1 (α 1)(1³ β(1 δ))´kt+1 = 0 b− − b− − − b b b b 7 b 3.2 Regressions as approximation

This type of approximation is particularly common in economics as it just corresponds to ordinary least square (OLS) . As you know the problem essentially amounts to approximate a function F by another function G of exogenous variables. In other words, we have a set of observable endogenous variables yi, i = 1 . . . N, which we are willing to explain in terms of the set of exogenous variables X = x1, . . . , xk , i = 1 . . . N. This problem amounts to ﬁnd a set i { i i } of parameters Θ that solves the problem

N min (y G(X ; Θ))2 (3.3) p i i Θ∈R − Xi=1 The idea is therefore that Θ is chosen such that on average G(X ; Θ) is close enough to y, such that G delivers a “good” approximation for the true function F in the region of x that we consider. This is actually nothing else than econometrics! There are however several choices that can be made and that give us much more freedom than in econometrics. We now consider investigate these choices.

Number of data points: Econometricians never have the choice of the points they can use to reveal information on the F function, as data are given by history. In numerical analysis this constraint is relaxed, we may be exactly identified for example, meaning that the number of data points, N, is exactly equal to the number of parameters, p, we want to reveal. Never would a good econometrician do this kind of thing! Obviously, we can impose a situation where N > p in order to exploit more information. The difference between these two choices should be clear to you, in the first case we are sure that the approximation will be exact in the selected points, whereas this will not necessarily be the case in the second experiment. To see that, just think of the following: assume we have a sample made of 2 data points for a function that we want to approximate using a linear function. we actually need 2 parameters

8 to deﬁne a linear function: the intercept, α and the slope β. In such a case, (3.3) rewrites 2 2 min (y1 α βx1) + (y2 α βx2) {α,β} − − − − which yields the system of orthogonality conditions

(y α βx ) + (y α βx ) = 0 1 − − 1 2 − − 2 (y α βx )x + (y α βx )x = 0 1 − − 1 1 2 − − 2 2 which rewrites 1 1 y α βx 1 − − 1 A.v = 0 x1 x2 y2 α βx2 ≡ µ ¶ µ − − ¶ This system then just amounts to ﬁnd the null space of the matrix A, which, in the case x = x (which we can always impose as we will see next), imposes 1 6 2 vi = 0, i = 1, 2. This therefore leads to

y1 = α + βx1

y2 = α + βx2 such that the approximation is exact. When the system is overidentiﬁed this is not the case anymore.

Selection of data points: This is actually a major difference between econometrics and numerical approximations. We do control the space over which we want to take an approximation. In particular, we can spread the data points wherever we want in order to control information. For instance, let us consider the particular case of the function depicted in figure (3.1). As this function exhibits a kink in x? it may be beneficial to concentrate a lot of points around x? in order to reveal as much information as possible on the kink.

Functional forms: One key issue in the selection of the approximating function is a functional form. In most of the cases, a sequence of monomials

9 Figure 3.1: Selection of points

y = F (x) 6

- x? x

of different degrees is selected: X = 1, x , x2, . . . , xp , One advantage of this i { i } selection is basically simplicity. However, this often turns to be a very bad choice as many problems then turn out to be ill–conditioned with such a choice. The most obvious objection we may formulate against this specification choice is related to the multicolinearity problem. Assume for instance that you want to approximate a production function that depends on employment and the capital stock. The capital stock is basically an extremely smooth variable as, by construction, it is essentially a smooth moving average of investment decisions ∞ k = i + (1 δ)k k = (1 δ)ì t+1 t − t ⇐⇒ t+1 − t−` X`=0 Therefore, taking as a basis for exogenous variables powers of the capital stock is basically a bad idea. To see that, let us assume that δ = 0.025 and it is a white noise process with volatility 0.1, then let us simulate a 1000 data points j process for the capital stock and let us compute the correlation matrix for kt ,

10 j = 1, . . . , 4, we get: 2 3 4 kt kt kt kt kt 1.0000 0.9835 0.9572 0.9315 2 kt 0.9835 1.0000 0.9933 0.9801 3 kt 0.9572 0.9933 1.0000 0.9963 4 kt 0.9315 0.9801 0.9963 1.0000 implying a very high correlation between the powers of the capital stock, therefore raising the possibility of occurrence of multicolinearity. A typical answer to this problem is to rely on orthogonal polynomials rather than monomials. We will discuss this issue extensively in the next section. A second possibility is to rely on parcimonious approaches that do not require too much information in terms of function specification. An alternative is to use neural networks. Before going to Neural Network approximations, we first have to deal with a potential problem you may face with all the examples I gave is that when we solve a model: the true decision rule is unknown, such that we do not know the function we are dealing with. However, the main properties of the decision rule are known, in particular, we know that it has to satisfy some conditions imposed by economic theory. As an example, let us attempt to find an approximate solution for the consumption decision rule in the deterministic optimal growth model. Economic theory teaches us that consumption should satisfy the Euler equation

c−σ = βc−σ αkα−1 + 1 δ (3.4) t t+1 t+1 − knowing that the capital stock evolves¡ as ¢

k = kα c + (1 δ)k (3.5) t+1 t − t − t Let us assume that consumption may be approximated by

2 φ(kt, θ) = exp θ0 + θ1 log(kt) + θ2 log(kt) over the interval [k; k]. Our problem¡ is then to ﬁnd the triple¢ θ , θ , θ such { 0 1 2} that N 2 φ(k , θ)−σ βφ(k , θ)−σ αkα−1 + 1 δ t − t+1 t+1 − t=1 X £ ¡ ¢¤ 11 is minimum. This actually amounts to solve a non–linear least squares problem. However, a lot of structure is put on this problem as kt+1 has to satisfy the law of motion for capital:

k = kα exp θ + θ log(k ) + θ log(k )2 + (1 δ)k t+1 t − 0 1 t 2 t − t ¡ ¢ The algorithm then works as follows

1. Set a grid of N data points, k N , for the capital stock over the interval { i}i=1 [k; k], and an initial vector θ , θ , θ . { 0 1 2} 2. for each k , i = 1, . . . , N, and given θ , θ , θ , compute i { 0 1 2}

ct = φ(kt, θ)

and k = kα φ(k , θ) + (1 δ)k t+1 t − t − t 3. Compute

ct+1 = φ(kt+1, θ)

and the quantity

R(k , θ) φ(k , θ)−σ βφ(k , θ)−σ αkα−1 + 1 δ t ≡ t − t+1 t+1 − ¡ ¢ 4. If the quantity N 2 R(ki, θ) t=1 X is minimal then stop, else update θ , θ , θ and go back to 2. { 0 1 2} As an example, I computed the approximate decision rule for the deterministic optimal growth model with α = 0.3, β = 0.95, δ = 0.1 and σ = 1.5. I consider that the model may deviate up to 90% from its capital stock steady state — k = 0.1k? and k = 1.9k? — and used 20 data points. Figure 3.2 reports the approximate decision rule versus the “true” decision rule.4 As can be seen

4The “true” decision rule was computed using value iteration, which we shall study later.

12 from the graph, the solution we obtained with our approximation is rather accurate and we actually have

N true LS true LS 1 ci ci −4 ci ci tr−ue = 8.743369e and max tr−ue = 0.005140 N c i={1,...,N} c i=1 ¯ i ¯ ¯ i ¯ X ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ such that¯ the error ¯we are making is particularly¯small. Indeed,¯ using the approximate decision rule, an agent would make a maximal error of 0.51% in its economic calculus — i.e. 51 cents for each 100$ spent on consumption.

Figure 3.2: Least–square approximation of consumption

1.8 True Decision Rule LS approximation 1.6

1.4

1.2

1 Consumption

0.8

0.6

0.4 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Capital stock

A neural network may be simply viewed as a particular type of function, which are flexible enough to fit fairly general functions. A neural network may be simply understood using the standard metaphor of human brain. There is an input, x, which is processed by a node. Each node is actually a function that transforms the input into an output which is then itself passed to another node. For instance, panel (a) of figure 3.3, borrowed from Judd [1998], illustrates the

13 single–layer neural network whose functional form is

n (i) G(x, θ) = h θig(x ) Ã ! Xi=1 where x is the vector of inputs, and h and g are scalar functions. A common choice for g in the case of the dingle layer neural network is g(x) = x. There- fore, if we set h to be identity too, we are back to the standard OLS model with monomials. A second a perhaps much more interesting type of neural

Figure 3.3: Neural Networks

1 1 x x q 1 2 2 x ~ x qR . q - y . -s - y . . 3 . > . . . U . . :w xn xn (a) (b) network is depicted in panel (b) of ﬁgure 3.3. This type of neural network is called the hidden–layer feedforward network. In this speciﬁcation, we are closer to the idea of network as the transformed input is fed to another node that process it to get information. The associated function form is given by

m n j (i) G(x, θ, γ) = f γjh θi g(x )  Ã ! Xj=1 Xi=1   In this case, h is called the hidden–layer activation function. h should be a “squasher” function — i.e. a monotically nondecreasing function that maps R onto [0; 1]. Three very popular functions are

1. The heaviside step function

1 for x 0 h(x) = > 0 for x < 0 ½

14 2. The sigmoid function 1 h(x) = 1 + exp( x) − 3. Cumulative distribution functions, for example the normal cdf 1 x x2 h(x) = exp √2πσ −2σ2 Z−∞ µ ¶ Obtaining an approximation then simply amounts to determine the set of coefficients γ , θj; i = 1 . . . n, j = 1, . . . , m . This can be simply achieved { j i } running non–linear least squares, that is solving N 2 min (y` G(x`, θ, γ)) {θ,γ} − X`=1 One particular nice feature of neural networks is that they deliver accurate approximation using only few parameters. This characteristic is related to the high flexibility of the approximating functions they use. Further, as established by the following theorem by Hornik, Stinchcombe and White [1989], neural network are extremely powerful in that they offer a universal approximation method.

Theorem 3 Let f : Rn R be a continuous function to be approximated. −→ ∞ Let h be a continuous function, h : R R, such that either (i) h(x)dx → −∞ is ﬁnite and non zero and h is Lp for 1 6 p 6 or (ii) h is a “squashing” ∞ R function (nondecreasing, limx→∞ h(x) = 1,limx→−∞ h(x) = 0). Let n n n j Σ (h) = g : R R, g(x) = θjh(w .x + aj), ( → Xj=1 j n j aj, θj R, and w R , w = 0, m = 1, 2, . . . ∈ ∈ 6 ) be the set of all possible single hidden–layer feedforward neural networks, using h as the hidden layer activation function. Then, for all ε > 0, probability measure µ, and compact sets K Rn, there is a g Σ(h) such that ⊂ ∈ sup f(x) g(x) 6 ε and f(x) g(x) dµ 6 ε | − | | − | x∈K ZK

15 This theorem is of great importance as it provides a universal approximation result that states that for a broad class of functions, neural networks deliver an accurate approximation to any continuous function. Indeed, we may use for instance any squashing function of the type we described above, or any simple function that satisﬁes condition (i). Nevertheless one potential limitation of the approach lies into the fact that we have to conduct non–linear estimation, which may be cumbersome under certain circumstances. As an example, we will deal with the function

3 1 3 F (x) = min max , x , 2 −2 − 2 Ã Ã µ ¶ ! ! over the interval [ 3; 3] and consider a single hidden–layer feedforward network − of the form

θ θ F (x, θ, ω, α) = 1 + 2 1 + exp( (ω x + α ) 1 + exp( (ω x + α ) − 1 1 − 2 2 The algorithme is then straightforward

1. Generate N values for x [ 3; 3] and compute F (x) ∈ − 2. Set initial values for Θ = θ , ω , α ; i = 1, 2 0 { i i i } 3. Compute N (F (x ) F (x , Θ))2 i − i Xi=1 if this quantity is minimal then stop, eelse update Θ and go back to 2.

The last step can be performed using a non–linear minimizer, such that we are actually performing non–linear least–squares. Figure 3.4 plots the approximation where N = 1000 and the solution vector Θ yields the values reported in table 3.2. Note that from the form of the Θ vector, we can deduce that the ﬁrst layer handle positive values for x, therefore corresponding to the left part of the function, while the second layer takes care of the negative part of the

16 Table 3.2: Neural Network approximation

θ1 ω1 α1 θ2 ω2 α2 2.0277 6.8424 -10.0893 -1.5091 -7.6414 -2.9238

function. Further, it appears that the function is no so badly approximated by this simple neural network, as

1 N 2 1 2 E2 = (F (xi) F (xi, Θ) = 0.0469 N Ã − ! Xi=1 e b and

E∞ = max F (xi) F (xi, Θ) = 0.2200 i − ¯ ¯ ¯ ¯ ¯ e b ¯

Figure 3.4: Neural Network Approximation 2.5 True 2 Neural Network

1.5

0.5

−0.5

−1

−1.5

−2 −3 −2 −1 0 1 2 3 x

All these methods actually relied on regressions. They are simple, but may be either totally unreliable and ill–conditioned in a number of problem, or

17 diﬃcult to compute because they rely on non–linear optimization. We will now consider methods which will actually be more powerful and somehow simpler to implement. They however require introducing some important preliminary concepts, among which that of orthogonal polynomials

3.2.1 Orthogonal polynomials

This class of polynomial possesses — by deﬁnition — the orthogonality property which will prove to be extremely eﬃcient and useful in a number of problem. For instance, this will solve the multicolinearity problem we encountered in OLS. This property will also greatly simplify the computation of the approximation in a number of problems we will deal with in the sequel. First of all we need to introduce some preliminary concepts.

Deﬁnition 4 (Weighting function) A weighting function ω(x) on the interval [a; b] is a function that is positive almost everywhere on [a; b] and has a ﬁnite integral on [a; b].

An example of such a weighting function is ω(x) = (1 x)−1/2 over the interval − 0 [-1;1]. Indeed, limx→−1 ω(x) = √2/2, and ω (x) > 0 such that ω(x) is positive everywhere over the whole interval. Further

1 1 (1 x2)−1/2dx = arcsin(x) = π − Z−1 ¯−1 ¯ ¯ Deﬁnition 5 (Inner product) Let us consider two¯ functions f1(x) and f2(x) both deﬁned at least on [a; b], the inner product with respect to the weighting function ω(x) is given by

b f , f = f (x)f (x)ω(x)dx h 1 2i 1 2 Za For example, assume that f (x) = 1, f (x) = x and ω(x) = (1 x2)( 1/2), 1 2 − − then the inner product over the interval [-1;1] is

1 1 x 2 f1, f2 = dx = 1 x = 0 h i √1 x − − Z−1 ¯−1 − p ¯ ¯ 18 ¯ Hence, in this case, we have that the inner product of 1 and x with respect to ω(w) on the interval [-1;1] is identically null. This will actually deﬁne the orthogonality property.

Deﬁnition 6 (Orthogonal Polynomials) The family of polynomials P (x) { n } is mutually orthogonal with respect to ω(x) iﬀ

P , P = 0 for i = j h i ji 6

Deﬁnition 7 (Orthonormal Polynomials) The family of polynomials P (x) { n } is mutually orthonormal with respect to ω(x) iﬀ it is orthogonal and

P , P = 1 for all i h i ii

Table 3.3 reports the most common families of orthogonal polynomials (see Judd [1998] for a more detailed exposition) and table 3.4 their recursive for- mulation

Table 3.3: Orthogonal polynomials (deﬁnitions)

Family ω(x) [a; b] Deﬁnition (−1)n dn 2 n Legendre 1 [ 1; 1] Pn(x) = 2nn! dxn (1 x ) 2 −1/2 − −1 − Chebychev (1 x ) [ 1; 1] Tn(x) = cos(n cos (x)) − − exp(x) dn n Laguerre exp( x) [0, ) Ln(x) = n! dxn (x exp( x)) − ∞ n − Hermite exp( x2) ( , ) H (x) = ( 1)n exp(x2) d exp( x2) − −∞ ∞ n − dxn −

3.3 Least square orthogonal polynomial approximation

We will now discuss one of the most common approach to approximation that goes back to the easy OLS approach we have seen previously.

19 Table 3.4: Orthogonal polynomials (recursive representation)

Family 0 1 Recursion Legendre P (x) = 1 P (x) = x P (x) = 2n+1 xP (x) n P (x) 0 1 n+1 n+1 n − n+1 n−1 Chebychev T (x) = 1 T (x) = x T (x) = 2xT (x) T (x) 0 1 n+1 n − n−1 Laguerre L (x) = 1 L (x) = 1 x L (x) = 2n+1−x L (x) n L (x) 0 1 − n+1 n+1 n − n+1 n−1 Hermite H (x) = 1 H (x) = 2x H (x) = 2xH (x) 2nH (x) 0 1 n+1 n − n−1

Deﬁnition 8 Let F : [a, b] R be a function we want to approximate, −→ and g(x) a polynomial approximation of F . The least square polynomial approximation of F with respect to the weighting function ω(w) is the degree n polynomial that solves

b min (F (x) g(x))2 ω(x)dx deg(g)6n − Za This equation may be understood adopting an econometric point of view, as ω(w) may be given a weighting matrix interpretation we ﬁnd in GMM estimation. Indeed, ω(x) furnishes an indication on how much we care about approximation errors as a function of x. Therefore, setting ω(x) = 1, which amounts to put the same weight on any x then corresponds to a simple OLS approximation. Let us now assume that g(x) = n c ϕ (x), where ϕ (x) n is a se- i=0 i i { k }k=0 quence of orthogonal polynomials, thePleast square problem rewrites 2 b n min F (x) ciϕi(x)) ω(x)dx {c }n i i=0 a Ã − ! Z Xi=0 for which the ﬁrst order conditions are given by

b n F (x) ciϕi(x)) ϕi(x)ω(x)dx = 0 for i = 0, . . . , n a Ã − ! Z Xi=0 20 which yields F, ϕ c = h ii i ϕ , ϕ h i ii Therefore, we have n F, ϕi F (x) h i ϕi(x) ' ϕi, ϕi Xi=0 h i There are several examples of the use of this type of approximation. Fourier approximation is an example of those which is suitable for periodic functions. Nevertheless, I will focus in a coming section on one particular approach, which we will use quite often in the next chapters: Chebychev approximation.

Beyond least square approximation, there exists other approaches that departs from the least square by the norm they use:

Uniform approximation, which attempt to solve •

lim max F (x) pn(x) = 0 n→∞ x∈[a;b] | − |

The main diﬀerence between this approach and L2 approximation is that contrary to L2 norms that put no restrictions on the approximation on particular points, the uniform approximation imposes that the approximation of F at each x is exact, whereas L2 approximations juste requires the total error to be as small as possible.

Minimax approximations, which rest on the L∞ norm, such that these • approximations attempt to ﬁnd an approximation that provides the best uniform approximation to the function F that is we search the degree n polynomial that achieves

min F (x) g(x) ∞ deg(g)6n k − k

There are theorems in approximation theory that states that the quality of this approximation increases polynomially as the degree of polynomials increases, and the polynomial rate of convergence is faster for

21 smoother functions. Nevertheless, if this approximation can be as good as needed for F , no use of its derivatives is made, such that the derivatives may be very poorly approximated. Finally, such approximation methods are extremely diﬃcult to compute.

3.4 Interpolation methods

Up to now, we have seen that there exist methods to compute the value of a function at some particular points, but in most of the cases we are not only interested by the function at some points but also between these points. This is the problem of interpolation.

3.4.1 Linear interpolation

This method is totally straightforward and known to everybody. Assume you have a collection of data points C = (x , y ) i = 1, . . . , n . Then for any { i i | } x [x , x ], we can compute y as ∈ i−1 i y y x y x y y = i − i−1 x + i i−1 − i−1 i x x x x i − i−1 i − i−1 Although, such approximations have proved to be very useful in a lot of applications, it can be particularly ineﬃcient for diﬀerent reasons:

1. It does not deliver an approximating function but rather a collection of approximations for each interval;

2. it requires to identify the interval where the approximation is to be computed, which may be particularly costly when the interval is not uniform;

3. it can obviously perform very badly for non–linear functions.

Therefore, there obviously exist alternative interpolation methods that perform better, but which are not necessarily more eﬃcient.

22 3.4.2 Lagrange interpolation

This type of approximation is quite demanding as it consider a collection of data C = (x , y ) i = 1, . . . , n with distinct x — called the Lagrange data. { i i | } i Lagrange interpolation amounts to ﬁnd a degree n 1 polynomial P (x), such − that yi = P (xi). Therefore, the method is exact for each point. Lagrange interpolating polynomials are deﬁned by

x xj Pi(x) = − xi xj Yj6=i −

Note that Pi(xi) = 1 and Pi(xj) = 0. The interpolation is then given by

n P (x) = yiPi(x) Xi=1 The obvious problem that we can immediately see from this formula is that if the number of point is high enough, this type of interpolation is totally untractable. Indeed, just to compute a single P (x) this already requires 2(n i − 1) substractions, n multiplications. Then this has to be constructed for the n data points to compute all needed Pi(x). Then to compute P (x) we need n additions and n multiplications. Therefore, this requires 3n2 operations! Therefore, one actually may actually attempt to compute directly:

n−1 i P (x) = αix Xi=0 which may be obtained by solving the linear system

2 n−1 y1 = α0 + α1x1 + α2x1 + . . . + αn−1x1 y = α + α x + α x2 + . . . + α xn−1  2 0 1 2 2 2 n−1 2  .  .  2 n−1 yn = α0 + α1xn + α2xn + . . . + αn−1xn  or  Aα = y

23 where α = α , α , . . . , α 0 and A is the so–called Vandermonde matrix for { 0 1 n−1} xi, i = 1, . . . , n 2 n−1 1 x1 x1 x1 2 · · · n−1 1 x2 x2 x2 A =  . . . ·.· · .  ......    1 x x2 xn−1   n n · · · n  If the Lagrange formula guaran tees the existence ofthe interpolation, the following theorem guarantees its uniqueness.

Theorem 4 Provided the interpolating points are distincts, there is a unique solution to the Lagrange interpolation problem.

The proof of the theorem may be sketched as follows. Assume that besides P (x), there exists another interpolating polynomial P ?(x) of degree at most n 1 that also interpolates the n data points. Then, by construction, P (x) − − P ?(x) is at most of degree n 1 and is zeros at all n nodes. But the only − polynomial of degree n 1 that has n zeros is 0, such that P (x) = P ?(x). − Here again, the method may be quite demanding from a computational point of view, as we have to invert a matrix of size (n n). There then exist much more × eﬃcient methods, and in particular the so–called Chebychev approximation that works very well for smooth functions.

3.5 Chebychev approximation

Chebychev approximation uses . . . Chebychev polynomials as a basis for the polynomials (see ﬁgure 3.5). These polynomials are described by the recursion

T (x) = 2xT (x) T (x) with T (x) = 1, T (x) = x n+1 n − n−1 0 1 which admits as solution

−1 Tn(x) = cos(n cos (x)) as aforementioned these polynomials form an orthogonal basis with respect to the weighting function ω(x) = (1 x2)−1/2 over the interval [ 1; 1]. Neverthe- − − less, this interval may be generalized to [a; b], by transforming the data using

24 the formula y a x = 2 − 1 for y [a; b] b a − ∈ − Beyond the standard orthogonality property, Chebychev polynomials exhibit a discrete orthogonality property which writes as

n 0 for i = j 6 T (r )T (r ) = n for i = j = 0 i k j k  i=1 n for i = j = 0 X  2 6 where rk, k = 1, . . . , n are the roots ofTn(x) = 0.

Figure 3.5: Chebychev polynomials 1 n=1 0.8 n=2 n=3 0.6 n=4

0.4

0.2

−0.2

−0.4

−0.6

−0.8

−1 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 x

A ﬁrst theorem will state the usefulness of Chebychev approximation.

Theorem 5 Assume F is a Ck function over the interval [ 1; 1], and let − 1 2 F (x)Ti(x) ci = dx for i = 0 . . . n 2 π −1 √1 x Z − and n c g (x) 0 + c T (x) n ≡ 2 i i Xi=1 25 Then, there exists ε < such that for all n > 2 ∞ log(n) F (x) gn(x) ∞ 6 ε k − k nk This theorem is of great importance as it actually states that the approximation gn(x) will be as close as we might want to F (x) as the degree of approximation n increases to . In eﬀect, since the approximation error is log(n) ∞ bounded by above by ε nk and since the latter expression tends to zero as n tends toward , we have that g (x) converges uniformly to F (x) as n ∞ n increases. Further, the next theorem will establish a useful property on the coeﬃcients of the approximation.

Theorem 6 Assume F is a Ck function over the interval [ 1; 1], and admits − a Chebychev representation ∞ c F (x) = 0 + c T (x) 2 i i Xi=1 then, there exists c such that c ci 6 for j > 1 | | ik This theorem is particularly important as it states that the smoother the function to be approximated is (the greater k is), the faster is the pace at which coefficients will drop off. In other words, we will be able to achieve a high enough accuracy using less coefficients.

At this point, we have established that Chebychev approximation can be accurate for smooth functions, but we still do not know how to proceed to get a good approximation. In particular, a very important issue is the selection of interpolating data points, the so–called nodes. This is the main problem of interpolation: how to select nodes such that we minimize the interpolation error? The answer to this question is particularly simple in the case of Cheby- chev interpolation: the nodes should be the zeros of the nth degree Chebychev polynomial.

26 We are then endowed with data points to compute the approximation. Using m > n data points, we can compute the (n 1)th order Chebychev − approximation relying on the Chebychev regression algorithm we will now describe in details. When m = n, the algorithm reduces to the so–called Cebychev interpolation formula. Let us consider the following problem: Let F : [a; b] R, let us construct a degree n 6 m polynomial approximation of −→ F on [a; b]: n x a G(x) α T 2 − 1 ≡ i i b a − Xi=0 µ − ¶ 1. Compute m > n + 1 Chebychev interpolation nodes on [ 1; 1], which − are the roots of the degree m Chebychev polynomial

2k 1 r = cos − π for k = 1 . . . , m k − 2m µ ¶

2. Adjust the nodes, rk, to ﬁt in the [a; b] interval b a x = (r + 1) − + a for k = 1 . . . , m k k 2

3. Evaluate the function F at each approximation node xk, to get a collection of ordinates

yk = F (xk) for k = 1 . . . , m

4. Compute the collection of n + 1 coeﬃcients α = α ; i = 0 . . . n as { i } m ykTi(rk) k=1 αi = Xm 2 Ti(rk) Xk=1 5. Form the approximation

n x a G(x) α T 2 − 1 ≡ i i b a − Xi=0 µ − ¶

27 Note that step 4 actually can be interpreted in terms of an OLS problem as

— because of the orthogonality property of the Chebychev polynomials — αi is given by cov(y, Ti(x)) αi = var(Ti(x)) which may be recast in matricial notations as

α = (X0X)−1X0Y where T (x ) T (x ) T (x ) y 0 1 1 1 · · · n 1 1 T0(x2) T1(x2) Tn(x2) y2 X =  . . ·.· · .  and Y =  .  ......      T (x ) T (x ) T (x )   y   0 m 1 m · · · n m   m     

We now report two examples implementing the algorithm. The first one deals with a smooth function of the type F (x) = xθ. The second one evaluate the accuracy of the approximation in the case of a non–smooth function: F (x) = min(max( 1.5, (x 1/2)3), 2). − − In the case of the smooth function, we set θ = 0.1 and approximate the function over the interval [0.01; 2]. We select 100 nodes and evaluate the accuracy of degree 2 and 6 approximation. Figure 3.6 reports the true function and the corresponding approximation, table 3.5 reports the coefficients. As can be seen from the table, adding terms in the approximation does not alter the coefficients of lower degree. This just reflects the orthogonality properties of the Chebychev polynomials, that we saw in the formula determining each

αi. This is of great importance, as it states that once we have obtained a high order approximation, obtaining lower orders is particularly simple. This is the economization principle. Further, as can be seen from the ﬁgure, a “good approximation” to the function is obtained at rather low degrees. Indeed, the diﬀerence between the function and its approximation at order 6 is already good.

28 Table 3.5: Chebychev Coeﬃcients: Smooth function

n=2 n=6 c0 0.9547 0.9547 c1 0.1567 0.1567 c2 -0.0598 -0.0598 c3 – 0.0324 c4 – -0.0202 c5 – 0.0136 c6 – -0.0096

Figure 3.6: Smooth function: F (x) = x0.1 Approximation Residuals 1.1 0.05

1 0 0.9 True n=2 −0.05 0.8 n=6 −0.1 0.7 n=2 n=6 0.6 −0.15 0 0.5 1 1.5 2 0 0.5 1 1.5 2 x x

29 Matlab Code: Smooth Function Approximation m = 100; % number of nodes n = 6; % degree of polynomials rk = -cos((2*[1:m]-1)*pi/(2*m)); % Roots of degree m polynomials a = 0.01; % lower bound of interval b = 2; % upper bound of interval xk = (rk+1)*(b-a)/2+a; % nodes Y = xk.^0.1; % compute the function at nodes % % Builds Chebychev polynomials % Tx = zeros(m,n+1); Tx(:,1) = ones(m,1); Tx(:,2) = xk(:); for i=3:n+1; Tx(:,i) = 2*xk(:).*Tx(:,i-1)-Tx(:,i-2); end % % Chebychev regression % alpha = X\Y; % compute the approximation coefficients G = X*a; % compute the approximation

In the case of the non–smooth function we consider (F (x) = min(max( 1.5, (x − − 1/2)3), 2)), the coeﬃcients remain large even at degree 15 and the residuals remain high at order 15. This actually indicates that Chebychev approximations are well suited for smooth functions, but have more diﬃculties to capture kinks. Nevertheless, increasing the order of the approximation drastically, we can achieve a much better approximation. In the later case, it seems that a piecewise approximation would perform better. Indeed, in this case, we may compute 3 approximation

for x ( , x), G(x) = 1.5 • ∈ −∞ − for x (x, x), G(x) n β T 2 x−a 1 • ∈ ≡ i=0 i i b−a − P ³ ´ for x (x, ), G(x) = 2 • ∈ ∞ where x is such that (x 1/2)3 = 1.5 − − 30 Table 3.6: Chebychev Coeﬃcients: Non–smooth function

n=3 n=7 n=15 c0 -0.0140 -0.0140 -0.0140 c1 2.0549 2.0549 2.0549 c2 0.4176 0.4176 0.4176 c3 -0.3120 -0.3120 -0.3120 c4 – -0.1607 -0.1607 c5 – -0.0425 -0.0425 c6 – -0.0802 -0.0802 c7 – 0.0571 0.0571 c8 – – 0.1828 c9 – – 0.0275 c10 – – -0.1444 c11 – – -0.0686 c12 – – 0.0548 c13 – – 0.0355 c14 – – -0.0012 c15 – – 0.0208

Figure 3.7: Non–smooth function: F (x) = min(max( 1.5, (x 1/2)3), 2) − − Approximation Residuals 3 1 True n=3 2 n=3 n=7 n=7 0.5 n=15 1 n=15

0 0 −1

−2 −0.5 −4 −2 0 2 4 −4 −2 0 2 4 x x

31 and x satisﬁes (x 1/2)3 = 2 − In such a case, the approximation would be perfect with n = 3. This suggests that piecewise approximation may be of interest in a number of cases.

3.6 Piecewise interpolation

We have actually already seen piecewise approximation method: the linear interpolation method. But there exist more powerful and eﬃcient method that use splines. A spline can be any smooth function that is piecewise polynomial, but most of all it should be smooth at all nodes.

Deﬁnition 9 A function S(x) on an interval [a; b] is a spline of order n if

1. S(x) is a C n−2 function on [a; b],

2. There exist a collection of ordered nodes a = x0 < x1 < . . . < xm = b such that S(x) is a polynomial of order n 1 on each interval [x ; x ], − i i+1 i = 0, . . . , m 1 −

Examples of spline functions are

Cubic splines: These splines functions are splines of order 4. These • splines are the most popular and are of the form

S (x) = a + b (x x ) + c (x x )2 + d (x x )3 for x [x ; x ] i i i − i i − i i − i ∈ i i+1

B0–splines: These functions are splines of order 1: •

0, x < xi 0 B (x) = 1, xi 6 x 6 xi+1 i   0, x > xi+1  32 B1–splines: These functions are splines of order 2 that actually describe • tent functions:

0, x < xi x−xi , xi 6 x 6 xi+1 1  xi+1−xi Bi (x) = xi+2−x  x −x , xi+1 6 x 6 xi+2  i+2 i+1 0, x > xi+2   Such a spline reaches apeak at x = xi+1 and is upward (downward)

sloping for x < xi+1 (x > xi+1).

Higher order spline functions are defined by the recursion: • x x x x Bn(x) = − i Bn−1(x) + i+n+1 − Bn−1(x) i x x i x x i+1 µ i+n − i ¶ µ i+n+1 − i+1 ¶ Cubic splines are the most widely used splines to interpolate functions. There- fore, we will describe the method in greater details in such a case. Let us assume that we are endowed with Lagrange data — i.e. a collection of nodes xi and corresponding values for the function yi = F (xi) to interpolate: (x , y ) : i = 0 . . . n . We therefore have in hand n intervals [x ; x ], { i i } i i+1 i = 0, . . . , n 1 for which we search n cubic splines − S (x) = a + b (x x ) + c (x x )2 + d (x x )3 for x [x ; x ] i i i − i i − i i − i ∈ i i+1 The problem is then to select 4n coefficients a , b , c , d : i = 0, . . . , n 1 { i i i i − } using n + 1 nodes. We therefore need 4n identification conditions to identify these 4n coefficients. The first set of restrictions is given by the collection of restrictions imposing that the spline approximation is exact on the nodes

S(x ) = y for i = 0, . . . , n 1 and S (x ) = y i i − n−1 n n which amounts to impose

a = y for i = 0, . . . , n 1 (3.6) i i − 33 and

a + b (x x ) + c (x x )2 + d (x x )3 = y (3.7) n−1 n−1 n − n−1 n−1 n − n−1 n−1 n − n−1 n The second set of restrictions imposes continuity of the function on the upper bound of each interval

S (x ) = S (x ) for i = 1, . . . , n 1 i i i−1 i − which implies, noting h = x x i i − i−1 a = a + b h + c h2 + d h3 for i = 1, . . . , n 1 (3.8) i i−1 i−1 i i−1 i i−1 i − This furnishes 2n identification restrictions, such that 2n additional restrictions are still needed. Since we are dealing with a cubic spline interpolation, this requires the approximation to be C 2, implying that first and second order derivatives should be continuous. This yields the following n 1 conditions − for the first order derivatives

S0(x ) = S0 (x ) for i = 1, . . . , n 1 i i i−1 i − or b = b + 2c h + 3d h2 for i = 1, . . . , n 1 (3.9) i i−1 i−1 i 3i−1 i − and the additional n 1 conditions for the second order derivatives − S00(x ) = S00 (x ) for i = 1, . . . , n 1 i i i−1 i − or 2c = 2c + 6d h for i = 1, . . . , n 1 (3.10) i i−1 3i−1 i − Equations (3.6)–(3.10) therefore deﬁne a system of 4n 2 equations, such that − we are left with 2 degrees of freedom. Hence, we have to impose 2 additional conditions. There are several ways to select such conditions

00 1. Natural cubic splines impose that the second order derivatives S0 (x0) = 00 Sn(xn) = 0. Note that the latter is actually not to be calculated in our

34 problem, nevertheless this imposes conditions on both c0 and cn which will be useful in the sequel. In fact it imposes

c0 = cn = 0

An interpretation of this condition is that the cubic spline is represented

by the tangent of S at x0 and xn

2. Another way to ﬁx S(x) would be to use potential information on the slope of the function to be approximated. In other words, one may set

0 0 0 0 S0(x0) = F (x0) and Sn−1(xn) = F (xn)

This is the so–called Hermite spline. However, in a number of situation such information on the derivative of F is either not known or does not exist (think of F not being diﬀerentiable at some points), such that further source of information is needed. One can then rely on an approximation of the slope by the secant line. This is what is proposed 0 by thesecant Hermite spline, which amounts to approximate F (x0) and 0 F (xn) by the secant line over the corresponding interval: S (x ) S (x ) S (x ) S (x ) S0 (x ) = 0 1 − 0 0 and S0 (x ) = n−1 n − n−1 n−1 0 0 x x n−1 n x x 1 − 0 n − n−1 But from the identiﬁcation scheme, we have S0(x1) = S1(x1) = y1 and

Sn−1(xn) = yn, such that we get

b = (y y )/h1 and b = (y y )/h 0 1 − 0 n−1 n − n−1 n

Let us now focus on the natural cubic spline approximation, which imposes c0 = cn = 0. First, note that the system (3.6)–(3.9) has a recursive form, such that from (3.9) we can get 1 di−1 = (ci ci−1) for i = 1, . . . , n 1 3hi − − Plugging this results in (3.9),we get

b b = 2c h + (c c )h = (c + c )h for i = 1, . . . , n 1 i − 1i−1 i−1 i i − i−1 i i i−1 i − 35 and, (3.8) becomes

1 a a = b h + c h2 + (c c )h2 i − i−1 i−1 i i−1 i 3 i − i−1 i 1 = b h + (c + 2c )h2 for i = 1, . . . , n 1 i−1 i 3 i i−1 i − which we may rewrite as

ai ai−1 1 − = bi−1 + (ci + 2ci−1)hi for i = 1, . . . , n 1 hi 3 − Likewise, we have

ai+1 ai 1 − = bi + (ci+1 + 2ci)hi+1 for i = 0, . . . , n 2 hi+1 3 − substracting the last two equations, when deﬁned, we get

ai+1 ai ai ai−1 1 1 − − = bi bi−1 + (ci+1 + 2ci)hi+1 (ci + 2ci−1)hi hi+1 − hi − 3 − 3 for i = 1, . . . , n 2, which is then given, taking (3.6) and (3.7) into account, − by

3 3 (yi+1 yi) (yi yi−1) = hici−1 + 2(hi + hi+1)ci + hi+1ci+1 hi+1 − − hi − for i = 1, . . . , n 2. We however have the additional n 1–th identiﬁcation − − restriction that imposes c0 = 0 and the last restriction cn = 0 We therefore end–up with a system of the form

Ac = B where

2(h0 + h1) h1 h1 2(h1 + h2) h2   h2 2(h2 + h3) h3 A =  .   ..     h 2(h + h ) h   n−3 n−3 n−2 n−2   h 2(h + h )   n−2 n−2 n−1    36 3 3 c (y2 y1) (y1 y0) 1 h1 − − h0 − . . c =  .  and B =  .  3 3 cn−1 (yn yn−1) (yn−1 yn−2)    hn−1 − − hn−2 −      The matrix A is then said to be tridiagonal (and therefore sparse) and is also symmetric and elementwise positive. It is hence positive deﬁnite and therefore invertible. We then got all the c , i = 1, . . . , n 1 and can compute the b’s i − and d’s as

yi yi−1 1 yn yn−1 2cn−1 bi−1 = − (ci+2ci−1)hi for i = 1, . . . , n 1 and bn−1 = − hi −3 − hn − 3hn and 1 cn−1 di−1 = (ci ci−1) for i = 1, . . . , n 1 and dn−1 = 3hi − − − 3hn finally we have had a = y , i = 0, . . . , n 1 from the very beginning. i i − Once the approximation is obtained, the evaluation of the approximation has to be undertaken. The only difficult part in this job is to identify the interval the value of the argument of the function we want to evaluate belongs to — i.e. we have to find i 0, . . . , n 1 such that x [x , x ]. Nevertheless, ∈ { − } ∈ i i+1 as long as the nodes are generated using an invertible formula, there will be no cost to determine the interval. Most of the time, a uniform grid is used, such that the interval [a; b] is divided using the linear scheme xi = a + i∆, where ∆ = (b a)/(n 1), and i = 0, . . . , n 1. In such a case, it is particularly − − − simple to determine the interval as i is given by E[(x a)/∆]. Nevertheless − there are some cases where it may be efficient to use non–uniform grid. For instance, in the case of the function we consider it would be useful to consider the following simple 4 nodes grid 3, 0.5 √3 1.5, 0.5 + √3 2, 3 , as taking this {− − } grid would yield a perfect approximation (remember that the central part of the function is cubic!) As an example of spline approximation, figure 3.8 reports the spline approximation to the non–smooth function F (x) = min(max( 1.5, (x 1/2)3), 2) − − considering a uniform grid over the [-3;3] interval with 3, 7 and 15 nodes. In

37 Figure 3.8: Cubic spline approximation Approximation Residuals 3 1.5 True n=3 2 n=3 1 n=7 n=7 n=15 1 n=15 0.5 0 0 −1

−2 −0.5 −4 −2 0 2 4 −4 −2 0 2 4 x x order to gauge the potential of spline approximation, we report in the upper panel of ﬁgure 3.9 the L2 and L∞ error of approximation. The L2 approximation error is given by F (x) S(x) while the L∞ is given by max F (x) S(x) . k − k | − | It clearly appears that increasing the number of nodes improves the approximation in that the error is driven to zero. Nevertheless, it also appears that convergence is not monotonic in the case of the L∞ error. This is actually related to the fact that F , in this case is not even C 1 on the overall interval. In fact, as soon as we consider a smooth function this convergence is monotonic, as can be seen from the lower panel that report it for the function F (x) = x0.1 over the interval [0.01;2]. This actually illustrates the following result.

4 Theorem 7 Let F be a C function over the interval [x0; xn] and S its cubic spline approximation on x0, x1, . . . , xn and let δ > maxi xi xi−1 , then { } { − }

5 (4) 4 F S ∞ 6 F ∞δ k − k 384k k and 0 0 9 + (3) (4) 3 F S ∞ 6 F ∞δ k − k 216p k k This theorem actually gives upper bounds to spline approximation, and indicates that these bounds decrease at a fast pace (power of 4) as the number of

38 Figure 3.9: Approximation errors

F (x) = min(max(−1.5, (x − 1/2)3), 2) over [−3; 3] L2 error L∞ error 0.03 0.8

0.6 0.02 0.4 0.01 0.2

0 0 0 20 40 60 0 20 40 60 # of nodes # of nodes F (x) = x0.1 over [0.01; 2]

−3 2 ∞ x 10 L error L error 3 0.2

0.15 2 0.1 1 0.05

0 0 0 20 40 60 0 20 40 60 # of nodes # of nodes

39 nodes increases (as δ diminishes). Splines are usually viewed as a particularly good approximation method for two main reasons:

1. A good approximation may be achieved even for functions that are not C ∞ or that do not possess high order derivatives. Indeed, as indicated in theorem 7, the error term basically depends only on fourth order derivatives, such that even if the ﬁfth order derivative were badly behaved then an accurate approximation may be obtained.

2. Evaluation of splines is particularly cheap as they involve most of the time at most cubic polynomials, the only costly part being the interval search step. Matlab Code: Cubic Spline Approximation nbx = 8; % number of nodes a = -3; % lower bound of interval b = 3; % upper bound of interval dx = (b-a)/(n-1); % step in the grid x = [a:dx:b]; % grid points y = min(max(-1.5,(x(i)-0.5)^3),2);

A = spalloc((nbx-2),(nbx-2),3*nbx-8); % creates sparse matrix A B = zeros((nbx-2),1); % creates vector B A(1,[1 2])=[2*(dx+dx) dx]; for i=2:nbx-3; A(i,[i-1 i i+1])=[dx 2*(dx+dx) dx]; B(i)=3*(y(i+2)-y(i+1))/dx-3*(y(i+1)-y(i))/dx; end A(nbx-2,[nbx-3 nbx-2])=[dx 2*(dx+dx)];

c = [0;A\B]; a = y(1:nbx-1); b = (y(2:nbx)-y(1:nbx-1))/dx-dx*([c(2:nbx-1);0]+2*c(1:nbx-1))/3; d = ([c(2:nbx-1);0]-c(1:nbx-1))/(3*dx); S = [a’;b’;c(1:nbx-1)’;d’]; % Matrix of spline coefficients

One potential problem that may arise with the type of method we have developed until now is that we have not imposed any particular restriction on the shape of the approximation relative to the true function. This may be of great importance in some cases. Let us assume for instance that we need to approximate the function F (xt) that characterizes the dynamics of variable x

40 in the following backward looking dynamic equation:

xt+1 = F (xt)

Assume F is a concave function that is costly to compute, such that it is bene- ﬁcial to approximate the function. However, as we have already seen from the previous examples, many methods generate oscillations in the approximation. This can create some important problems as it implies that the approximation is not strictly concave, which is in turn crucial to characterize the dynamics of variable x. Further, the approximation of a strictly increasing function may be locally decreasing. All this may create some divergent path, or even generate some spurious steady state, and therefore spurious dynamics. It is therefore crucial to develop shape preserving methods — preserving in particular the curvature and monotonicity properties — for such cases.

3.7 Shape preserving approximations

In this section, we will see an approximation method that preserves the shape of the function we want to approximate. This method was proposed by Schu- maker [1983] and essentially amounts to exploit some information on both the level and the slope of the function to be approximated to build a smooth approximation. We will deal with two situations. The ﬁrst one — Hermite interpolation — assumes that we have information on both the level and the slope of the function to approximate. The second one — that uses Lagrange data — assumes that no information on the slope of the function is available. Both method was originally developed using quadratic splines.

3.7.1 Hermite interpolation

This method assumes that we have information on both the level and the slope of the function to be approximated. Assume we want to approximate 0 the function F on the interval [x1, x2] and we know yi = F (xi) and zi = F (xi), i = 1, 2. Schumaker proposes to build a quadratic function S(x) on [x1; x2]

41 that satisﬁes 0 S(xi) = yi and S (xi) = zi for i = 1, 2 Schumaker establishes ﬁrst that

Lemma 1 If z + z y y 1 2 = 2 − 1 2 x x 2 − 1 then the quadratic form z z S(x) = y + z (x x ) + 2 − 1 (x x )2 1 1 − 1 2(x x ) − 1 2 − 1 0 satisﬁes S(xi) = yi and S (xi) = zi for i = 1, 2.

The construction of this function is rather appealing. If z1 and z2 have the 0 same sign then S (x) has the same sign as z1 and z2 over [x1; x2]: (z z ) S0(x) = z + 2 − 1 (x x ) 1 (x x ) − 1 2 − 1 Hence, if F is monotically increasing (decreasing) on the interval [x1; x2], so is

S(x). Further, z1 > z2 (z1 < z2) indicates concavity (convexity), which S(x) satisﬁes as S00(x) = (z z )/(x x ) < 0 (> 0). 2 − 1 2 − 1 However, the conditions stated by this lemma are extremely stringent and do not usually apply, such that we have to adapt the procedure. This may be done by adding a node between x1 and x2 and construct another spline that satisﬁes the lemma.

Lemma 2 For every x? (x , x ) there exist a unique quadratic spline that ∈ 1 2 solves 0 S(xi) = yi and S (xi) = zi for i = 1, 2 with a node at x?. This spline is given by α + α (x x ) + α (x x )2 for x [x ; x?] S(x) = 01 11 − 1 21 − 1 ∈ 1 α + α (x x?) + α (x x?)2 for x [x?; x ] ½ 02 12 − 22 − ∈ 2 where z−z1 α = y α = z α = ? 01 1 11 1 21 2(x −x1) z+z1 ? z2−z α02 = y1 + (x x1) α12 = z α22 = ? 2 − 2(x2−x ) ? ? where z = 2(y2−y1)−(z1(x −x1)+z2(x2−x ) x2−x1

42 If the later lemma fully characterized the quadratic spline, it gives no information on x? which therefore remains to be selected. x? will be set such that the spline matches the desired shape properties. First note that if z1 and z2 are both positive (negative), then S(x) is monotone if and only if z1z > 0 (6 0) which is actually equivalent to

? ? 2(y2 y1) R (x x1)z1 + (x2 x )z2 if z1, z2 R 0 − − − This essentially deals with the monotonicity problem, and we now have to tackle the question of curvature. To do so, we compute the slope of the secant line between x1 and x2 y y ∆ = 2 − 1 x x 2 − 1 Then, if (z2 ∆)(z1 ∆) > 0, this indicates the presence of an inﬂexion point − − in the interval [x1; x2] such that the interpolant cannot be neither convex nor concave. Conversely, if z ∆ < z ∆ and x? satisﬁes | 2 − | | 1 − |

? 2(x2 x1)(z2 ∆) x1 < x 6 x x1 + − − ≡ (z z ) 2 − 1 then S(x), as described in the latter lemma, is convex (concave) if z1 < z2

(z1 > z2). Further, if z1z2 > 0 it is also monotone. If, on the contrary, z ∆ > z ∆ and x? satisﬁes | 2 − | | 1 − |

2(x2 x1)(z1 ∆) ? x x2 + − − 6 x < x2 ≡ (z z ) 2 − 1 then S(x), as described in the latter lemma, is convex (concave) if z1 < z2

(z1 > z2). This therefore endow us with a range of values for x? that will insure that shape properties will be preserved.

? 1. Check if lemma 1 is satisﬁed. If so set x = x2 and set S(x) as in lemma 2. Then stop else go to 2.

2. Compute ∆ = y y /x x 2 − 1 2 − 1 43 ? 3. if (z1 ∆)(z2 ∆) > 0 set x = (x1 + x2)/2 and stop else goto 4. − − 4. if z ∆ < z ∆ set x? = (x + x)/2 and stop else goto 5. | 1 − | | 2 − | 1 ? 5. if z1 ∆ > z2 ∆ set x = (x2 + x)/2 and stop. | − | | − | ? We have then in hand a value for x for [x1; x2]. We then apply it to each sub–interval to get x? [x ; x ] and then solve the general interpolation i ∈ i i+1 problem as explained in lemma 2. Note here that everything assumes that with have Hermite data in hand — i.e. x , y , z : i = 0, . . . , n . However, the knowledge of the slope is usually { i i i } not the rule and we therefore have to adapt the algorithm to such situations.

3.7.2 Unknown slope: back to Lagrange interpolation

Assume now that we do not have any data for the slope of the function, that is we are only endowed with Lagrange data x , y : i = 0, . . . , n . In such a { i i } case, we just have to add the needed information — an estimate of the slope of the function — and proceed exactly as in Hermite interpolation. Schumaker proposes the following procedure to get z ; i = 1, . . . , n . Compute { i } 1 L = (x x )2 + (y y )2 2 i i+1 − i i+1 − i £ ¤ and y y ∆ = i+1 − i i x x i+1 − i for i = 1, . . . , n 1. Then z , i = 1, . . . , n can be recovered as − i L ∆ + L ∆ i−1 i−1 i i if ∆ ∆ > 0 z = L + L i−1 i i = 2, . . . , n 1 i  i−1 i −  0 if ∆i−1∆i 6 0 and  3∆ z 3∆ s z = 1 − 2 and z = n−1 − n−1 1 − 2 n 2 Then, we just apply exactly the same procedure as described in the previous section.

44 Up to now, all methods we have been studying are uni–dimensional whereas most of the model we deal with in economics involve more than 1 variable. We therefore need to extend the analysis to higher dimensional problems.

3.8 Multidimensional approximations

Computing a multidimensional approximation to a function may be quite cumbersome and even impossible in some cases. To understand the problem, let us restate an example provided by Judd [1998]. Consider we have data points 2 P1, P2, P3, P4 = (1, 0), ( 1, 0), (0, 1), (0, 1) in R and the corresponding { } { − − } data zi = F (Pi), i = 1, . . . , 4. Assume now that we want to construct the approximation of function F using a linear combination of 1, x, y, xy deﬁned { } as G(x, y) = a + bx + cy + dxy such that G(xi, yi) = zi. Finding a, b, c, d amounts to solve the linear system

1 1 0 0 a z1 1 1 0 0 b z2  −    =   1 0 1 0 c z3  1 0 1 0   d   z   −     4  which is not feasibleas the matrix is notfull rank.  This example reveals two potential problems:

1. Approximation in higher dimensional systems involves cross–product and therefore poses the problem of the selection of polynomial basis to be used for approximation,

2. More important is the selection of the grid of nodes used to evaluate the function to compute the approximation.

We now investigate these issues, by ﬁrst considering the simplest way to attack the question — namely considering tensor product bases — and then moving to a second way of dealing with this problem — considering complete polynomials. In each case, we explain how Chebychev approximations can be obtained.

45 3.8.1 Tensor product bases

The idea here is to use the tensor product of univariate functions to form a basis of multivariate functions. In order to better understand this point, let us consider that we want to approximate a function F : R2 R using simple −→ univariate monomials up to order 2: X = 1, x, x2 and Y = 1, y, y2 . The { } { } tensor product basis is given by

1, x, y, xy, x2, y2, x2y, xy2, x2y2 { } i.e. all possible 2–terms products of elements belonging to X and Y . We are now in position to deﬁne the n–fold tensor product basis for functions of n variables x , . . . , x , . . . , x . { 1 i n}

Deﬁnition 10 Given a basis for n functions of the single variable xi: Pi = pk(x ) κi then the tensor product basis is given by { i i }k=0

κ1 κn B = . . . pk1 (x ) . . . pkn (x )  1 1 n n  kY1=0 kYn=0  An important problem with this type of tensor product basis is their size. For example, considering a m–dimensional space with polynomials of order n, we already get (n + 1)m terms! This exponential growth in the number of terms makes it particularly costly to use this type of basis, as soon as the number of terms or the number of nodes is high. Nevertheless, it will often be satisfactory or suﬃcient for low enough polynomials (in practice n=2!) Therefore, one often rely on less computationally costly basis.

3.8.2 Complete polynomials

As aforementioned, tensor product bases grow exponentially as the dimension of the problem increases, complete polynomials have the great advantage of growing only polynomially as the dimension increases. From an intuitive point of view, complete polynomials bases take products of order lower than a priori given κ into account, ignoring higher terms of higher degrees.

46 Deﬁnition 11 For κ N given, the complete set of polynomials of total degree ∈ κ in n variables is given by

n Bc k1 kn = x1 . . . xn : k1, . . . , kn > 0, ki 6 κ ( × × ) Xi=1 To see this more clearly, let us consider the example developed in the previous section (X = 1, x, x2 and Y = 1, y, y2 ) and let us assume that κ = 2. In { } { } this case, we end up with a complete polynomials basis of the type

Bc = 1, x, y, x2, y2, xy = B xy2, x2y, x2y2 \{ } © ª Note that we have actually already encountered this type of basis, as this is typically what is done by Taylor’s theorem for many dimensions

n ? ∂F ? ? F (x) F (x ) + (x )(xi xi ) ' ∂xi − Xi=1 . . n n 1 ∂F ? ? ? + . . . (x )(xi1 xi1 ) . . . (xik xik ) k! ∂xi1 . . . ∂xi1 − − iX1=1 iXk=1 For instance, considering the Taylor expansion to the 2–dimensional function F (x, y) around (x?, y?) we get

F (x, y) F (x?, y?) + F (x?, y?)(x x?) + F (x?, y?)(y y?) ' x − y − 1 ? ? ? 2 ? ? ? ? + Fxx(x , y )(x x ) + 2Fxy(x , y )(x x )(y y ) 2Ã − − −

? ? ? 2 +Fyy(x , y )(y y ) − ! which rewrites

2 2 F (x, y) = α0 + α1x + α2y + α3x + α4y + α5xy such that the implicit polynomial basis is the complete polynomials basis of order 2 with 2 variables.

47 The key diﬀerence between tensor product bases and complete polynomials bases lies essentially in the rate at which the size of the basis increases. As aforementioned, tensor product bases grow exponentially while complete polynomials bases only grow polynomially. This reduces the computational cost of approximation. But what do we loose using complete polynomials rather than tensor product bases? From a theoretical point of view, Taylor’s theorem gives us the answer: Nothing! Indeed, Taylor’s theorem indicates that the element in Bc delivers a approximation in the neighborhood of x? that exhibits an asymptotic degree of convergence equal to k. The n–fold tensor product, B, can deliver only a kth degree of convergence as it does not contains all terms of degree k + 1. In other words, complete polynomials and tensor product bases deliver the same degree of asymptotic convergence and therefore complete polynomials based approximation yields an as good level of accuracy as tensor product based approximations. Once we have chosen a basis, we can proceed to approximation. For example, we may use Chebychev approximation in higher dimensional problems. Judd [1998] reports the algorithm for this problem. As we will see, it takes advantage of a very nice feature of orthogonal polynomials: they inherit their orthogonality property even if we extend them to higher dimensions. Let us then assume we want to compute the chebychev approximation of a 2–dimensional function F (x, y) over the interval [a ; b ] [a ; b ] and let us assume — to x x × y y keep things simple for a while — that we use a tensor product basis. Then the algorithm is as follows

1. Choose a polynomial order for x (nx) and y (ny)

2. Compute mx > nx + 1 and my > ny + 1 Chebychev interpolation nodes on [ 1; 1] − 2k 1 zx = cos − π , k = 1, . . . , m k 2m x µ x ¶ and 2k 1 zy = cos − π , k = 1, . . . , m k 2m y µ y ¶

48 3. Adjust the nodes to ﬁt in both interval

b a x = a + (1 + zx) x − x , k = 1, . . . , m k x k 2 x µ ¶ and b a y = a + (1 + zy) y − y , k = 1, . . . , m k y k 2 y µ ¶ 4. Evaluate the function F at each node to form

Ω ω = F (x , y ) : k = 1, . . . , m ; ` = 1, . . . , m ≡ { k` k ` x y}

5. Compute the (n +1) (n +1) Chebychev coeﬃcients α , i = 0, . . . , n , x × y ij x j = 0, . . . , ny as

mx my x x y y ωk`Ti (zk ) Tj z` k=1 `=1 αij = X X mx my ¡ ¢ x x 2 y y 2 Ti (zk ) Tj z` Ãk=1 ! Ã`=1 ! X X ¡ ¢ which may be simply obtained in this case as

T x(zx)0ΩT y(zy) α = T x(zx) 2 T y(zy) 2 k k × k k 6. Compute the approximation as

nx ny x x ax y y ay G(x, y) = αijTi 2 − 1 Tj 2 − 1 bx ax − by ay − Xi=0 Xj=0 µ − ¶ µ − ¶ which may also be obtained as

x a y a 0 G(x, y) = T x 2 − x 1 α T y 2 − y 1 b a − b a − µ x − x ¶ µ y − y ¶ As an illustration of the algorithm we compute the approximation of the CES function 1 F (x, y) = [xρ + yρ] ρ

49 on the [0.01; 2] [0.01; 2] interval for ρ = 0.75. We used 5–th order polynomials × for both x and y and 20 nodes for both x and y, such that there are 400 possible interpolation nodes. Applying the algorithm we just described, we get the matrix of coefficients reported in table 3.7. As can be seen from the table, most of the coefficients are close to zero as soon as they involve the cross–product of higher order terms, such that using a complete polynomial basis would yield the same efficiency at a lower computational cost. Figure 3.10 reports the graph of the residuals for the approximation.

Table 3.7: Matrix of Chebychev coeﬃcients (tensor product basis)

k k 0 1 2 3 4 5 x \ y 0 2.4251 1.2744 -0.0582 0.0217 -0.0104 0.0057 1 1.2744 0.2030 -0.0366 0.0124 -0.0055 0.0029 2 -0.0582 -0.0366 0.0094 -0.0037 0.0018 -0.0009 3 0.0217 0.0124 -0.0037 0.0016 -0.0008 0.0005 4 -0.0104 -0.0055 0.0018 -0.0008 0.0004 -0.0003 5 0.0057 0.0029 -0.0009 0.0005 -0.0003 0.0002

2 Matlab Code: Chebychev Coefficients in R (Tensor Product Basis) rho = 0.75; mx = 20; my = 20; nx = 5; ny = 5; ax = 0.01; bx = 2; ay = 0.01; by = 2; % % Step 1 % rx = cos((2*[1:mx]’-1)*pi/(2*mx)); ry = cos((2*[1:my]’-1)*pi/(2*my)); % % Step 2 % x = (rx+1)*(bx-ax)/2+ax; y = (ry+1)*(by-ay)/2+ay;

50 % % Step 3 % Y = zeros(mx,my); for ix=1:mx; for iy=1:my; Y(ix,iy) = (x(ix)^rho+y(iy)^rho)^(1/rho); end end % % Step 4 % Xx = [ones(mx,1) rx]; for i=3:nx+1; Xx= [Xx 2*rx.*Xx(:,i-1)-Xx(:,i-2)]; end Xy = [ones(my,1) ry]; for i=3:ny+1; Xy= [Xy 2*ry.*Xy(:,i-1)-Xy(:,i-2)]; end T2x = diag(Xx’*Xx); T2y = diag(Xy’*Xy); a = (Xx’*Y*Xy)./(T2x*T2y’);

Figure 3.10: Residuals: Tensor product basis

0.01

0.005

−0.005

−0.01

−0.015 0

0.5

1 0 1.5 0.5 1 1.5 2 2 y x

51 If we now want to perform the same approximation using a complete polynomials basis, we just have to modify the algorithm to take into account the fact that when iterating on i and j we want to impose i + j 6 κ. Let us compute is for κ = 5. This implies that the basis will consists of

x y x y x y x y x y 1, T1 (.), T1 (.), T2 (.), T2 (.), T3 (.), T3 (.), T4 (.), T4 (.), T5 (.), T5 (.), x y x y x y x y T1 (.)T1 (.), T1 (.)T2 (.), T1 (.)T3 (.), T1 (.)T4 (.), x y x y x y T2 (.)T1 (.), T2 (.)T2 (.), T2 (.)T3 (.), x y x y T3 (.)T1 (.), T3 (.)T2 (.), x y T4 (.)T1 (.)

Table 3.8: Matrix of Chebychev coeﬃcients (Complete polynomials basis)

k k 0 1 2 3 4 5 x \ y 0 2.4251 1.2744 -0.0582 0.0217 -0.0104 0.0057 1 1.2744 0.2030 -0.0366 0.0124 -0.0055 – 2 -0.0582 -0.0366 0.0094 -0.0037 – – 3 0.0217 0.0124 -0.0037 – – – 4 -0.0104 -0.0055 – – – – 5 0.0057 – – – – –

A first thing to note is that the coefficients that remain are the same as the one we got in the tensor product basis. This should not be any surprise as what we just find is just the expression of the Chebychev economization we already encountered in the uni–dimensional case and which is just the direct consequence of the orthogonality condition of chebychev polynomials. Figure 3.11 report the residuals from the approximation using the complete basis. As can be seen from the figure, this “constrained” approximation yields quantita- tively similar results compared to the tensor product basis, therefore achieving almost the same accuracy while being less costly from a computational point of view. In the matlab code section, we just report the lines in step 4 that are affected by the adoption of the complete polynomials basis.

52 Matlab Code: Complete Polynomials Specificities a=zeros(nx+1,ny+1); for ix=1:nx+1; iy = 1; while ix+iy-2<=kappa a(ix,iy)=(Xx(:,ix)’*Y*Xy(:,iy))./(T2x(ix)*T2y(iy)); iy=iy+1; end end

Figure 3.11: Residuals: Complete polynomials basis

0.02

0.01

−0.01

−0.02 0 0.5 0 0.5 1 1 1.5 1.5 2 2 x y

3.9 Finite element approximations

Finite element are extremely popular among engineers (especially in aero- nautics). This approach considers elements that are zero over most of the domain of approximation. Although they are extremely powerful in the case of 2–dimensional problems, they are more diﬃcult to implement in higher dimensions. We therefore focus on the bi–dimensional case.

53 3.9.1 Bilinear approximations

A bilinear interpolation proposes to interpolate data linearly in both coordi- nate directions. Assume that we have the values of a function F (x, y) at the four points P = ( 1, 1) P = ( 1, 1) 1 − − 2 − P = (1, 1) P = (1, 1) 3 − 4 A cardinal interpolation basis on [ 1; 1] [ 1; 1] is provided by the set of − × − functions b (x, y) = 1 (1 x)(1 y) b (x, y) = 1 (1 x)(1 + y) 1 4 − − 2 4 − b (x, y) = 1 (1 + x)(1 y) b (x, y) = 1 (1 + x)(1 + y) 3 4 − 4 4 All functions b are zero on all P , i = j, but on the point to which is associated i j 6 the same index (i = j). Therefore, an approximation of F (x, y) on [ 1; 1] − × [ 1; 1] is given by − F (x, y) F ( 1, 1)b (x, y)+F ( 1, 1)b (x, y)+F (1, 1)b (x, y)+F (1, 1)b (x, y) ' − − 1 − 2 − 3 4 If we have data on [a ; b ] [a ; b ], we use the linear transformation we have x x × y y already encountered a great number of times x a y a 2 − x 1, 2 − y 1 b a − b a − µ x − x y − y ¶ Then, if we have Lagrange data of the type x , y , z : i = 1, . . . , n , we { i i i } proceed as follows

1. Construct a grid of nodes for x and y;

2. Construct the interpolant over each square applying the previous scheme;

3. Piece all interpolant together.

Note that an important issue of piecewise interpolation is related to the continuity of the approximation: individual pieces must meet continuously at common edges. In bilinear interpolation, this is insured by the fact that two interpolants overlap only at the edges of rectangles on which the approximation is a linear interpolant of 2 common end points. This would not be

54 insured if we were to construct bi–quadratic or bi–cubic interpolations for example. Note that this type of interpolation scheme is typically what is done when a computer draw a 3d graph of a function. In ﬁgure 3.12, we plot the residuals of the bilinear approximation of the CES function we approximated in the previous section, with 5 uniform intervals (6 nodes such that x = 0.010, 0.408, 0.806, 1.204, 1.602, 2.000 and y = 0.010, 0.408, 0.806, 1.204, 1.602, 2.000 ). { } { } Like in the spline approximation procedure, the most diﬃcult step once we have obtained an approximation is to determine the square the point for which we want an approximation belongs to. We therefore face exactly the same type of problems.

Figure 3.12: Residuals of the bilinear approximation

0.5

−0.5

−1 0 2 0.5 1.5 1 1 1.5 0.5 2 0 y x

3.9.2 Simplicial 2D linear interpolation

This method essentially amounts to consider triangles rather than rectangles as an approximation basis. The idea is then to build triangles in the x y − − plane. To do so, and assuming that the lagrange data have already been

55 transformed using the linear map described earlier, we set 3 points

P1 = (0, 0), P2 = (0, 1), P3 = (1, 0) to which we associate 3 functions

b (x, y) = 1 x y, b (x, y) = y, b (x, y) = x 1 − − 2 3 which are such that all functions b are zero on all P , i = j, but on the point i j 6 to which is associated the same index (i = j). b1 b2 and b3 are the cardinal functions on P1, P2, P3. Let us now add the point P4 = (1, 1), then we have the following cardinal functions for P2, P3, P4:

b (x, y) = 1 x, b (x, y) = 1 y, b (x, y) = x + y 1 4 − 5 − 6 −

Therefore, on the square P1, P2, P3, P4 the interpolant for F is given by

F (0, 0)(1 x y) + F (0, 1)y + F (1, 0)x if x + y 6 1 − − G(x, y) =  F (0, 1)(1 x) + F (1, 0)(1 y) + F (1, 1)(x + y 1) if x + y > 1  − − − It should be clear to you that if these methods are pretty easy to implement in dimension 3, it becomes quite cumber some in higher dimensions, and noth that much is proposed in the literature, are most of these methods were de- signed by engineers and physicists that essentially have to deal with 2, 3 at most 4 dimensional problems. We will see that this will be a limitation in a number of economic applications.

56 Bibliography

Hornik, K., M. Stinchcombe, and H. White, Multi–Layer Feedforward Net- works are Universal Approximators, Neural Networks, 1989, 2, 359–366.

Judd, K.L., Numerical methods in economics, Cambridge, Massachussets: MIT Press, 1998.

Schumaker, L.L., On Shape–preseving Quadratic Spline Interpolation, SIAM Journal of Numerical Analysis, 1983, 20, 854–864.

57 Index

Analytic function, 4 Tensor product bases, 46

Chebychev approximation, 24 Vandermonde matrix, 23 Complete polynomials, 47 complete polynomials, 46

Economization principle, 28

Hidden–layer feedforward, 14

Least square orthogonal polynomial approximation, 19 Linearization, 6 Local approximations, 2 Log–linearization, 7

Multidimensional approximation, 45

Neural networks, 11

Ordinary least square, 8 Orthogonal polynomials, 18

Radius of convergence, 3

Single–layer, 14 Splines, 32

Taylor series expansion, 2

58 Contents

3 Approximation Methods 1 3.1 Local approximations ...... 2 3.1.1 Taylor series expansion ...... 2 3.2 Regressions as approximation ...... 8 3.2.1 Orthogonal polynomials ...... 18 3.3 Least square orthogonal polynomial approximation ...... 19 3.4 Interpolation methods ...... 22 3.4.1 Linear interpolation ...... 22 3.4.2 Lagrange interpolation ...... 23 3.5 Chebychev approximation ...... 24 3.6 Piecewise interpolation ...... 32 3.7 Shape preserving approximations ...... 41 3.7.1 Hermite interpolation ...... 41 3.7.2 Unknown slope: back to Lagrange interpolation . . . . . 44 3.8 Multidimensional approximations ...... 45 3.8.1 Tensor product bases ...... 46 3.8.2 Complete polynomials ...... 47 3.9 Finite element approximations ...... 53 3.9.1 Bilinear approximations ...... 53 3.9.2 Simplicial 2D linear interpolation ...... 55

59 60 List of Figures

3.1 Selection of points ...... 10 3.2 Least–square approximation of consumption ...... 13 3.3 Neural Networks ...... 14 3.4 Neural Network Approximation ...... 17 3.5 Chebychev polynomials ...... 25 3.6 Smooth function: F (x) = x0.1 ...... 29 3.7 Non–smooth function: F (x) = min(max( 1.5, (x 1/2)3), 2) . 31 − − 3.8 Cubic spline approximation ...... 38 3.9 Approximation errors ...... 39 3.10 Residuals: Tensor product basis ...... 52 3.11 Residuals: Complete polynomials basis ...... 54 3.12 Residuals of the bilinear approximation ...... 56

61 62 List of Tables

3.1 Taylor series expansion for log(1 x) ...... 5 − 3.2 Neural Network approximation ...... 17 3.3 Orthogonal polynomials (definitions) ...... 19 3.4 Orthogonal polynomials (recursive representation) ...... 20 3.5 Chebychev Coefficients: Smooth function ...... 29 3.6 Chebychev Coefficients: Non–smooth function ...... 31 3.7 Matrix of Chebychev coefficients (tensor product basis) . . . . 50 3.8 Matrix of Chebychev coefficients (Complete polynomials basis) 52