Numerical

Carlos Hurtado

Department of Economics University of Illinois at Urbana-Champaign [email protected]

Sep 19th, 2017

C. Hurtado (UIUC - Economics) Numerical Methods On the Agenda

1 Numerical Python

2 Solving Systems of Linear Equations

3 LU Decomposition

4 Cholesky

5 Accuracy of Operations

C. Hurtado (UIUC - Economics) Numerical Methods Numerical Python On the Agenda

1 Numerical Python

2 Solving Systems of Linear Equations

3 LU Decomposition

4 Cholesky Factorization

5 Accuracy of Operations

C. Hurtado (UIUC - Economics) Numerical Methods Numerical Python Numerical Python

I The NumPy package (read as NUMerical PYthon) provides access to a new data structure called arrays which allow us to perform efficient vector and operations.

I NumPy is the updated version of two previous modules: Numeric and Numarray.

I In 2006 it was decided to merge the best aspects of Numeric and Numarray into the Scientific Python (scipy) package and to provide an array data type under the module name NumPy.

I NumPy contains some linear algebra functions.

C. Hurtado (UIUC - Economics) Numerical Methods 1 / 39 Numerical Python Numerical Python

I NumPy introduces a new data type which is called ”array” I An array appears to be very similar to a list but an array can keep only elements of the same type (arrays are more efficient to store)

I Vectors and matrices are all called ”arrays” in NumPy. I To create a Vector (one dimensional array) we do: 1 >>> import numpy as np 2 >>>x= np.array ([0 , 0.5 , 1 , 1.5]) 3 [0 , 0.5 , 1 , 1.5]

I We can also creation a vector using ”ArrayRange” 1 x=np.arange(0,2,0.5) 2 [0 , 0.5 , 1 , 1.5]

C. Hurtado (UIUC - Economics) Numerical Methods 2 / 39 Numerical Python Numerical Python

I There are some useful functions for arrays: 1 >>>y=np.zeros(4) 2 [ 0. 0. 0. 0.] I Remember, we need to be aware of the reference 1 >>>z=y 2 >>>y[0]=99 3 >>>y[2]=-1.2 4 >>> print z 5 [99. 0. -1.2 0.] I Sometimes it is better to work with a copy of the object: 1 >>>z=y.copy() 2 >>>z[0]=0 3 >>>z[2]=0 4 >>> print z 5 [99. 0. -1.2 0.] 6 >>> print y 7 [ 0. 0. 0. 0.] C. Hurtado (UIUC - Economics) Numerical Methods 3 / 39 Numerical Python Numerical Python

I we can perform calculations on every element in the vector with a single statement: 1 >>>x+10 2 array([ 10. , 10.5, 11. , 11.5]) 3 >>>x**2 4 array([ 0. , 0.25, 1. , 2.25]) 5 >>> 2*x 6 array([ 0., 1., 2., 3.])

I To create a matrix we use a list of lists: 1 >>>X=np.array([[1,2],[3,4]]) 2 [[1 2] 3 [3 4]]

C. Hurtado (UIUC - Economics) Numerical Methods 4 / 39 Numerical Python Numerical Python

I There are several useful functions: 1 >>>Y=np.zeros((3,3)) 2 [[ 0. 0. 0.] 3 [ 0. 0. 0.] 4 [ 0. 0. 0.]] 5 >>>Z=np.ones((2,2)) 6 [[ 1. 1.] 7 [ 1. 1.]] 8 >>>I=np.identity(3) 9 [[ 1. 0. 0.] 10 [ 0. 1. 0.] 11 [ 0. 0. 1.]] I We can get the dimension of the matrix: 1 >>>A=np.array([[1 , 2 , 3] , [4 , 5 , 6]]) 2 [[1 2 3] 3 [4 5 6]] 4 >>>A.shape 5 (2, 3) C. Hurtado (UIUC - Economics) Numerical Methods 5 / 39 Numerical Python Numerical Python

I Individual elements can be accessed using the standard syntaxes.

I It is also possible to recover all the elements form a row or a column 1 >>>A[:,1] 2 [2 5] 3 >>>A[0,:] 4 [1 2 3]

I We can also transform arrays to list 1 >>> list (A[1,:]) 2 [4, 5, 6]

C. Hurtado (UIUC - Economics) Numerical Methods 6 / 39 Solving Systems of Linear Equations On the Agenda

1 Numerical Python

2 Solving Systems of Linear Equations

3 LU Decomposition

4 Cholesky Factorization

5 Accuracy of Operations

C. Hurtado (UIUC - Economics) Numerical Methods Solving Systems of Linear Equations Solving Systems of Linear Equations

I We can multiply matrices and vectors using the dot product function 1 >>>A=np.random.rand(5,5) 2 >>>x=np.random.rand(5) 3 >>>b=np.dot(A,x)

I To solve a system of equations A · x = b (given in matrix form) we can use the linear algebra package 1 >>> x2=np.linalg.solve(A,b)

I How does the computer solve a system of equations?

C. Hurtado (UIUC - Economics) Numerical Methods 7 / 39 LU Decomposition On the Agenda

1 Numerical Python

2 Solving Systems of Linear Equations

3 LU Decomposition

4 Cholesky Factorization

5 Accuracy of Operations

C. Hurtado (UIUC - Economics) Numerical Methods LU Decomposition LU Decomposition

I Consider a system of equations Ax = b. I Suppose that we can factorize A into two matrices, L and U, where L is a lower and U is an upper triangular matrix. I The system is then Ax = b LUx = b L Ux = b |{z} y Ly = b

I We could perform a 2-step solution for the system: 1. Solve the lower triangular system Ly = b, by forward substitution. 2. Solve the upper triangular system Ux = y, by back substitution.

C. Hurtado (UIUC - Economics) Numerical Methods 8 / 39 LU Decomposition LU Decomposition

I If we have a system with lower triangular matrix       l11 0 ··· 0 x1 b1        l21 l22 ··· 0   x2   b2   . .  ·  .  =  .   . .   .   .   . .   .   .  ln1 ln2 ··· lnn xn bn

I The solution can be computed as:

b1 x1 = l11

Pk−1 bk − j=1 lkj xj xk = lkk for k = 2, 3, ··· , n

C. Hurtado (UIUC - Economics) Numerical Methods 9 / 39 LU Decomposition LU Decomposition

I A similar method can be used to solve an upper triangular system (but starting form the last equation x = bn ). n unn

I The total number of divisions is n, the number of multiplications and additions is n(n − 1)/2 (why?).

2 I For a big n, the total number of operations is close to n /2

I That means that the time it takes to solve a system of equations increases in a quadratic proportion to the number of variables in the system

I This is know as quadratic time computation.

C. Hurtado (UIUC - Economics) Numerical Methods 10 / 39 LU Decomposition LU Decomposition

I Any nonsingular matrix A can be decomposed into two matrices L and U. I Let us consider  1 1 1    A =  2 3 5  4 6 8

I We would like to transform the matrix to get an upper triangular matrix. - Leave the first row unchanged. - replace the second row: multiply the first row by -2 and add to the second row - replace the third row: multiply the first row by -4 and add to the third row

C. Hurtado (UIUC - Economics) Numerical Methods 11 / 39 LU Decomposition LU Decomposition

I The previous can be computed as follows:  1 0 0   1 1 1   1 1 1        L1A =  −2 1 0   2 3 5  =  0 1 3  −4 0 1 4 6 8 0 2 4

I Now we would like to remove the 2 located at the third row and second column. - Leave the first and second rows unchanged. - replace the third row: multiply the second row by -2 and add to the third row  1 0 0   1 1 1   1 1 1        L2 (L1A) =  0 1 0   0 1 3  =  0 1 3  = U 0 −2 1 0 2 4 0 0 −2

C. Hurtado (UIUC - Economics) Numerical Methods 12 / 39 LU Decomposition LU Decomposition

I So we can write L2L1A = U where  1 0 0   1 0 0      L1 =  −2 1 0  , L2 =  0 1 0  −4 0 1 0 −2 1 and  1 1 1   1 1 1      A =  2 3 5  , U =  0 1 3  4 6 8 0 0 −2

−1 −1 I Then we can write A = L2 L1 U

C. Hurtado (UIUC - Economics) Numerical Methods 13 / 39 LU Decomposition LU Decomposition

I Notice that:  1 0 0  −1   L1 =  2 1 0  4 0 1 and  1 0 0  −1   L2 =  0 1 0  0 2 1

I The multiplication of lower triangular matrices is also a lower triangular matrix. In particular:

 1 0 0  −1 −1   L = L2 L1 =  2 1 0  4 2 1

C. Hurtado (UIUC - Economics) Numerical Methods 14 / 39 LU Decomposition LU Decomposition

I In summary:  1 1 1   1 0 0   1 1 1        A =  2 3 5  =  2 1 0   0 1 3  = LU 4 6 8 4 2 1 0 0 −2

I Notice that L is a unit lower triangular matrix, i.e., it has ones on the diagonal.

I What should we do for a general matrix A?

C. Hurtado (UIUC - Economics) Numerical Methods 15 / 39 LU Decomposition LU Decomposition

I Let us start with   a11 a12 ··· a1n    a21 a22 ··· a2n  A =  . . . .   . . .. .   . . .  an1 an2 ··· ann

1 I Suppose that a11 6= 0. Define li1 = ai1/a11, for i = 2, ··· , n. I Then:     1 1 1  0 0 ··· 0 a11 a12 ··· a1n   l1 0 ··· 0   0 a2 ··· a2    21   22 2n  (2) L1A = I −  . . . .  A =  . . . .  ≡ A   . . .. .   . . .. .    . . .   . . .  1 n n ln1 0 ··· 0 0 an2 ··· ann

C. Hurtado (UIUC - Economics) Numerical Methods 16 / 39 LU Decomposition LU Decomposition

I Proceeding column by column in similar fashion, we can construct a series of lower triangular matrices that replaces the elements below the diagonal with zeros. k (k) I Let us denote by aij the elements of the k − th matrix A . k I If akk 6= 0, we define

 ak  ij for j = k, i = k + 1, ··· , n k ak lij = kk  0 otherwise

and

( ak − lk ak for i = k + 1, ··· , n and j = k, ··· , n ak+1 = ij ik kj ij k aij otherwise

C. Hurtado (UIUC - Economics) Numerical Methods 17 / 39 LU Decomposition LU Decomposition

I We have just defined a sequence of matrices such that   0 ··· 0 0 ··· 0    ......    ......     (k+1)   k  (k) A = I −  0 ··· 0 lk+1,k ··· 0  A      ......    ......  k 0 ··· 0 lnk ··· 0

I In this way we can rewrite the matrix A as the multiplication LU.

C. Hurtado (UIUC - Economics) Numerical Methods 18 / 39 LU Decomposition LU Decomposition

I How does it work in practice? I Let us consider the system Ax = b:       1 1 1 x1 3        2 3 5   x2  =  2  4 6 8 x3 1

I We can rewrite the system as:         1 0 0 1 1 1 x1 3          2 1 0   0 1 3   x2  =  2  4 2 1 0 0 −2 x3 1

Let us define I       y1 1 1 1 x1        y2  =  0 1 3   x2  y3 0 0 −2 x3

C. Hurtado (UIUC - Economics) Numerical Methods 19 / 39 LU Decomposition LU Decomposition

I We can first solve       1 0 0 y1 3        2 1 0   y2  =  2  4 2 1 y3 1

The solution is y1 = 3, y2 = −4 and y3 = −3.

I Then we solve:       1 1 1 x1 3        0 1 3   x2  =  −4  0 0 −2 x3 −3

17 3 The solution of the system is then x1 = −10, x2 = − 2 and x3 = 2 .

C. Hurtado (UIUC - Economics) Numerical Methods 20 / 39 LU Decomposition LU Decomposition

I The proposed method for LU decomposition assumes that akk 6= 0. I What can we do to decompose the following matrix?  0 1 2     2 0 3  1 1 1

I We can change (pivot) first and second row pre-multiplying by

 0 1 0     1 0 0  0 0 1

Then proceed as before.

C. Hurtado (UIUC - Economics) Numerical Methods 21 / 39 LU Decomposition LU Decomposition

I Another point to consider has to do with the rounding error. If the elements of the diagonal are small, the fraction aij is big, and I akk the significant digits are removed.

I Consider for example the matrix " # 10−8 2 3 4

I If we compute 3/10E − 8 we get 30000000.0, ”chopping away” the significant digits.

I To avoid this we can pivot the rows.

C. Hurtado (UIUC - Economics) Numerical Methods 22 / 39 LU Decomposition LU Decomposition

I In practice, the LU factorization in python uses pivoting. I To perform the decomposition we use the lu function from the module linalg in scipy. 1 >>> from numpy import array 2 >>> from scipy.linalg import lu 3 >>>A= array([[0,1,2],[2,0,3],[1,1,1]]) 4 >>>P,L,U= lu(A)

I The previous code will generate the matrix L and U, but also a pivoting matrix P.

I The matrix A can be recover as P · LU 1 >>> from numpy import matmul 2 >>> print matmul(P,matmul(L,U))

C. Hurtado (UIUC - Economics) Numerical Methods 23 / 39 LU Decomposition LU Decomposition

I Let us consider another example 1 >>>B= array([[10E-8,2],[3,4]]) 2 >>>P,L,U= lu(B) 3 >>> print P 4 >>> print matmul(P,matmul(L,U))

I Notice the and the product P · LU 1 [[ 0. 1.] 2 [ 1. 0.]] and 1 [[ 1.00000000e-07 2.00000000e+00] 2 [ 3.00000000e+00 4.00000000e+00]]

C. Hurtado (UIUC - Economics) Numerical Methods 24 / 39 Cholesky Factorization On the Agenda

1 Numerical Python

2 Solving Systems of Linear Equations

3 LU Decomposition

4 Cholesky Factorization

5 Accuracy of Operations

C. Hurtado (UIUC - Economics) Numerical Methods Cholesky Factorization Cholesky Factorization

I There is a special case of the LU decomposition for a subset of square matrices

0 I A matrix is symmetric if A = A I A matrix A is positive semidefinite if: - A is symmetric and x 0Ax ≥ 0 for all x 6= 0

I If the previous inequality is strict, we call A a positive definite matrix. I Examples: " # " # " # 9 6 9 6 9 6 A = , A = , A = 1 6 5 2 6 4 3 6 3

C. Hurtado (UIUC - Economics) Numerical Methods 25 / 39 Cholesky Factorization Cholesky Factorization

I A1 is positive definite: " #" # 0 9 6 x1 2 2 2 2 x A1x = [x1x2] = 9x1 +12x1x2+5x2 = (3x1+2x2) +x2 6 5 x2

I A2 is positive semidefinite (but not positive definite): " #" # 0 9 6 x1 2 2 2 x A2x = [x1x2] = 9x1 + 12x1x2 + 4x2 = (3x1 + 2x2) 6 4 x2

I A3 is not positive semidefinite: " #" # 0 9 6 x1 2 2 2 2 x A3x = [x1x2] = 9x1 +12x1x2+3x2 = (3x1+2x2) −x2 6 3 x2

C. Hurtado (UIUC - Economics) Numerical Methods 26 / 39 Cholesky Factorization Cholesky Factorization

0 I More Examples: for a given matrix X, A = X X is positive semidefinite

v 0Av = v 0X 0Xv = (Xv)0(Xv) = kXvk2

I Some properties of a positive definite matrix A: - The diagonal elements of A are positive  0  a11 A21 0 - If we rewrite A = , the matrix A22 − (1/a11)A21A21 is A21 A22 also positive definite.  0   1 0  h 0 i a A − A v 1 0 11 21 a11 21 Hint: − a A21v v > 0 11 A21 A22 v

C. Hurtado (UIUC - Economics) Numerical Methods 27 / 39 Cholesky Factorization Cholesky Factorization

I Every positive definite matrix A can be factored as A = LL0

where L is lower triangular with positive diagonal elements.

I L is called the Cholesky factor of A

I L can be interpreted as the ’square root’ of a positive definite matrix

C. Hurtado (UIUC - Economics) Numerical Methods 28 / 39 Cholesky Factorization Cholesky Factorization

0 I We want to partition the matrix A = LL as " 0 # " #" 0 # a11 A21 l11 0 l11 L21 = 0 A21 A22 L21 L22 0 L22 " 2 0 # l11 l11L21 = 0 0 l11L21 L21L21 + L22L22

The elements of the diagonal of a positive definite matrix are positive, I √ so l11 = a11 is well defined and positive. I We can define L21 = (1/l11)A21 I Finally, we can compute the Cholesky factorization defined by 0 0 L22L22 = A22 − L21L21 1 0 = A22 − A21A21 a11

C. Hurtado (UIUC - Economics) Numerical Methods 29 / 39 Cholesky Factorization Cholesky Factorization

I Example       25 15 −5 l11 0 0 l11 l21 l31        15 18 0  =  l21 l22 0   0 l22 l32  −5 0 11 l31 l32 l33 0 0 l33

I First column of L  25 15 −5   5 0 0   5 3 −1         15 18 0  =  3 l22 0   0 l22 l32  −5 0 11 −1 l32 l33 0 0 l33

I second column of L " # " # " # " # 18 0 3 18 0 9 −3 − [3 − 1] = − 0 11 −1 0 11 −3 1 " # 9 3 = 3 10 C. Hurtado (UIUC - Economics) Numerical Methods 30 / 39 Cholesky Factorization Cholesky Factorization

I second column of L " # " #" # 9 3 3 0 3 1 = 3 10 1 l33 0 l33

2 2 I third column of L: 10 − 1 = l33 =⇒ l33 = 3

I In conclusion:  25 15 −5   5 0 0   5 3 −1         15 18 0  =  3 3 0   0 3 1  −5 0 11 −1 1 3 0 0 3

C. Hurtado (UIUC - Economics) Numerical Methods 31 / 39 Cholesky Factorization Cholesky Factorization

I How can we use the Cholesky Factorization? I Let us assume that there is a stock market with returns that are normally distributed with µ = 5% and σ2 = 10% I If we want to simulate the returns of the stock we can try the following: - Define the returns as r = µ + σZ, where Z ∼ N(0, 1) is a Standard Normal random variable.

I The linear combination that defines r simulates the stock:

E[r] = E[µ + σZ] = E[µ] + E[σZ] = µ

and 2 2 2 2 Var(r) = E[(r − µ) ] = E[σ Z ] = σ

C. Hurtado (UIUC - Economics) Numerical Methods 32 / 39 Cholesky Factorization Cholesky Factorization

I In a more realistic situation we would like to simulate the returns of a portfolio with more than one asset. I Let us start with a portfolio of two assets (you can generalize to any number of assets). I If the proportion invested on the first asset is λ, the return of the portfolio is r = λr1 + (1 − λ)r2

where r1 is the return of the first asset and r2 is the return of the second asset. I It may be the case that the returns of the assets are correlated. For example, if the first asset is stock, and the second asset is U.S. Treasuries, the returns will exhibit strong negative correlation (Why?)

I We need to account for the covariance of r1 and r2

C. Hurtado (UIUC - Economics) Numerical Methods 33 / 39 Cholesky Factorization Cholesky Factorization

I Let us assume that E[r1] = µ1 and E[r2] = µ2. The expected return of the portfolio is

E[r] = E[λr1 + (1 − λ)r2] = λE[r1] + (1 − λ)E[r2] = λµ1 + (1 − λ)µ2

I The variance of the portfolio is: 2 Var(r) = E[(r − λµ1 − (1 − λ)µ2) ] 2 = E[(λ(r1 − µ1) + (1 − λ)(r2 − µ2)) ] 2 2 2 2 = E[λ (r1 − µ1) + (1 − λ) (r2 − µ2) + 2λ(1 − λ)(r1 − µ1)(r2 − µ2)] 2 2 2 2 = λ E[(r1 − µ1) ] + (1 − λ) E[(r2 − µ2) ] + 2λ(1 − λ)E[(r1 − µ1)(r2 − µ2)] 2 2 = λ Var(r1) + (1 − λ) Var(r2) + 2λ(1 − λ)Cov(r1, r2)

I Also computable as " #" # h i Var(r ) Cov(r , r ) λ Var(r) = λ (1 − λ) 1 1 2 Cov(r1, r2) Var(r2) 1 − λ

C. Hurtado (UIUC - Economics) Numerical Methods 34 / 39 Cholesky Factorization Cholesky Factorization

I This matrix is known as the variance-covariance matrix: " # Var(r ) Cov(r , r ) Σ = 1 1 2 Cov(r1, r2) Var(r2)

I The variance-covariance matrix is symmetric, with all the elements on the diagonal positive and positive definite. I Notice that the we could generate a vector of random variables (a.k.a. −→ −→ multivariate normal distribution) x = (r1, r2), with µ = (µ1, µ2) and variance-covariance matrix Σ using the ’square root’ of Σ. That is, the Cholesky factorization s.t. Σ = LL0 I That is: " # " # " # r1 µ1 z1 = + L , where z1 ∼ N(0, 1) and z2 ∼ N(0, 1) r2 µ2 z2

C. Hurtado (UIUC - Economics) Numerical Methods 35 / 39 Accuracy of Operations On the Agenda

1 Numerical Python

2 Solving Systems of Linear Equations

3 LU Decomposition

4 Cholesky Factorization

5 Accuracy of Operations

C. Hurtado (UIUC - Economics) Numerical Methods Accuracy of Operations Accuracy of Operations

I We want to find the solution of Ax = b. I Suppose that, using some algorithm, we have computed a numerical solutionx ˆ

I We would like to be able to evaluate the absolute error kx − xˆk, or the relative error kx − xˆk kxk

I We don’t know the error, but we would like to find an upper bound I We begin analyzing the residual r = b − Axˆ

C. Hurtado (UIUC - Economics) Numerical Methods 36 / 39 Accuracy of Operations Accuracy of Operations

I How does the residual r relate to the error inx ˆ? r = b − Axˆ = Ax − Axˆ = A(x − xˆ)

I We have then x − xˆ = A−1r

I We can define the norm of a matrix as follows kAk = max kAxk s.t. kxk = 1 x

I Using that definition of norm of matrix it es easy to show that

−1 −1 kx − xˆk = A r ≤ A krk

−1 I This gives a bound on the absolute error inx ˆ in terms of A C. Hurtado (UIUC - Economics) Numerical Methods 37 / 39 Accuracy of Operations Accuracy of Operations

I Usually the relative error is more meaningful. I Using the definition of norm of a matrix we know that kbk ≤ kAk kxk I The previous implies that 1 kAk ≤ kxk kbk

I Hence, we have an upper bound for the relative error. kx − xˆk kAk ≤ A−1 krk kxk kbk

−1 I We are going to call the condition number cond(A) = kAk A

C. Hurtado (UIUC - Economics) Numerical Methods 38 / 39 Accuracy of Operations Accuracy of Operations

I How big can the relative error be? For a matrix A we have 1 krk kx − xˆk krk ≤ ≤ cond(A) cond(A) kbk kxk kbk

I If the condition number is close to 1, then the relative error and relative residual will be close I The accuracy of the solution depends on the conditioning number of the matrix. I If a matrix is ill-conditioned, then a small roundoff error can have a drastic effect on the output I If the matrix is well-conditioned, then the computerized solution is quite accurate. I The condition number is a property of the matrix A.

C. Hurtado (UIUC - Economics) Numerical Methods 39 / 39