Numerical Linear Algebra Carlos Hurtado Department of Economics University of Illinois at Urbana-Champaign [email protected] Sep 19th, 2017 C. Hurtado (UIUC - Economics) Numerical Methods On the Agenda 1 Numerical Python 2 Solving Systems of Linear Equations 3 LU Decomposition 4 Cholesky Factorization 5 Accuracy of Operations C. Hurtado (UIUC - Economics) Numerical Methods Numerical Python On the Agenda 1 Numerical Python 2 Solving Systems of Linear Equations 3 LU Decomposition 4 Cholesky Factorization 5 Accuracy of Operations C. Hurtado (UIUC - Economics) Numerical Methods Numerical Python Numerical Python I The NumPy package (read as NUMerical PYthon) provides access to a new data structure called arrays which allow us to perform efficient vector and matrix operations. I NumPy is the updated version of two previous modules: Numeric and Numarray. I In 2006 it was decided to merge the best aspects of Numeric and Numarray into the Scientific Python (scipy) package and to provide an array data type under the module name NumPy. I NumPy contains some linear algebra functions. C. Hurtado (UIUC - Economics) Numerical Methods 1 / 39 Numerical Python Numerical Python I NumPy introduces a new data type which is called ”array” I An array appears to be very similar to a list but an array can keep only elements of the same type (arrays are more efficient to store) I Vectors and matrices are all called ”arrays” in NumPy. I To create a Vector (one dimensional array) we do: 1 >>> import numpy as np 2 >>>x= np.array ([0 , 0.5 , 1 , 1.5]) 3 [0 , 0.5 , 1 , 1.5] I We can also creation a vector using ”ArrayRange” 1 x=np.arange(0,2,0.5) 2 [0 , 0.5 , 1 , 1.5] C. Hurtado (UIUC - Economics) Numerical Methods 2 / 39 Numerical Python Numerical Python I There are some useful functions for arrays: 1 >>>y=np.zeros(4) 2 [ 0. 0. 0. 0.] I Remember, we need to be aware of the reference 1 >>>z=y 2 >>>y[0]=99 3 >>>y[2]=-1.2 4 >>> print z 5 [99. 0. -1.2 0.] I Sometimes it is better to work with a copy of the object: 1 >>>z=y.copy() 2 >>>z[0]=0 3 >>>z[2]=0 4 >>> print z 5 [99. 0. -1.2 0.] 6 >>> print y 7 [ 0. 0. 0. 0.] C. Hurtado (UIUC - Economics) Numerical Methods 3 / 39 Numerical Python Numerical Python I we can perform calculations on every element in the vector with a single statement: 1 >>>x+10 2 array([ 10. , 10.5, 11. , 11.5]) 3 >>>x**2 4 array([ 0. , 0.25, 1. , 2.25]) 5 >>> 2*x 6 array([ 0., 1., 2., 3.]) I To create a matrix we use a list of lists: 1 >>>X=np.array([[1,2],[3,4]]) 2 [[1 2] 3 [3 4]] C. Hurtado (UIUC - Economics) Numerical Methods 4 / 39 Numerical Python Numerical Python I There are several useful functions: 1 >>>Y=np.zeros((3,3)) 2 [[ 0. 0. 0.] 3 [ 0. 0. 0.] 4 [ 0. 0. 0.]] 5 >>>Z=np.ones((2,2)) 6 [[ 1. 1.] 7 [ 1. 1.]] 8 >>>I=np.identity(3) 9 [[ 1. 0. 0.] 10 [ 0. 1. 0.] 11 [ 0. 0. 1.]] I We can get the dimension of the matrix: 1 >>>A=np.array([[1 , 2 , 3] , [4 , 5 , 6]]) 2 [[1 2 3] 3 [4 5 6]] 4 >>>A.shape 5 (2, 3) C. Hurtado (UIUC - Economics) Numerical Methods 5 / 39 Numerical Python Numerical Python I Individual elements can be accessed using the standard syntaxes. I It is also possible to recover all the elements form a row or a column 1 >>>A[:,1] 2 [2 5] 3 >>>A[0,:] 4 [1 2 3] I We can also transform arrays to list 1 >>> list (A[1,:]) 2 [4, 5, 6] C. Hurtado (UIUC - Economics) Numerical Methods 6 / 39 Solving Systems of Linear Equations On the Agenda 1 Numerical Python 2 Solving Systems of Linear Equations 3 LU Decomposition 4 Cholesky Factorization 5 Accuracy of Operations C. Hurtado (UIUC - Economics) Numerical Methods Solving Systems of Linear Equations Solving Systems of Linear Equations I We can multiply matrices and vectors using the dot product function 1 >>>A=np.random.rand(5,5) 2 >>>x=np.random.rand(5) 3 >>>b=np.dot(A,x) I To solve a system of equations A · x = b (given in matrix form) we can use the linear algebra package 1 >>> x2=np.linalg.solve(A,b) I How does the computer solve a system of equations? C. Hurtado (UIUC - Economics) Numerical Methods 7 / 39 LU Decomposition On the Agenda 1 Numerical Python 2 Solving Systems of Linear Equations 3 LU Decomposition 4 Cholesky Factorization 5 Accuracy of Operations C. Hurtado (UIUC - Economics) Numerical Methods LU Decomposition LU Decomposition I Consider a system of equations Ax = b. I Suppose that we can factorize A into two matrices, L and U, where L is a lower triangular matrix and U is an upper triangular matrix. I The system is then Ax = b LUx = b L Ux = b |{z} y Ly = b I We could perform a 2-step solution for the system: 1. Solve the lower triangular system Ly = b, by forward substitution. 2. Solve the upper triangular system Ux = y, by back substitution. C. Hurtado (UIUC - Economics) Numerical Methods 8 / 39 LU Decomposition LU Decomposition I If we have a system with lower triangular matrix l11 0 ··· 0 x1 b1 l21 l22 ··· 0 x2 b2 . · . = . . . . . . . ln1 ln2 ··· lnn xn bn I The solution can be computed as: b1 x1 = l11 Pk−1 bk − j=1 lkj xj xk = lkk for k = 2, 3, ··· , n C. Hurtado (UIUC - Economics) Numerical Methods 9 / 39 LU Decomposition LU Decomposition I A similar method can be used to solve an upper triangular system (but starting form the last equation x = bn ). n unn I The total number of divisions is n, the number of multiplications and additions is n(n − 1)/2 (why?). 2 I For a big n, the total number of operations is close to n /2 I That means that the time it takes to solve a system of equations increases in a quadratic proportion to the number of variables in the system I This is know as quadratic time computation. C. Hurtado (UIUC - Economics) Numerical Methods 10 / 39 LU Decomposition LU Decomposition I Any nonsingular matrix A can be decomposed into two matrices L and U. I Let us consider 1 1 1 A = 2 3 5 4 6 8 I We would like to transform the matrix to get an upper triangular matrix. - Leave the first row unchanged. - replace the second row: multiply the first row by -2 and add to the second row - replace the third row: multiply the first row by -4 and add to the third row C. Hurtado (UIUC - Economics) Numerical Methods 11 / 39 LU Decomposition LU Decomposition I The previous can be computed as follows: 1 0 0 1 1 1 1 1 1 L1A = −2 1 0 2 3 5 = 0 1 3 −4 0 1 4 6 8 0 2 4 I Now we would like to remove the 2 located at the third row and second column. - Leave the first and second rows unchanged. - replace the third row: multiply the second row by -2 and add to the third row 1 0 0 1 1 1 1 1 1 L2 (L1A) = 0 1 0 0 1 3 = 0 1 3 = U 0 −2 1 0 2 4 0 0 −2 C. Hurtado (UIUC - Economics) Numerical Methods 12 / 39 LU Decomposition LU Decomposition I So we can write L2L1A = U where 1 0 0 1 0 0 L1 = −2 1 0 , L2 = 0 1 0 −4 0 1 0 −2 1 and 1 1 1 1 1 1 A = 2 3 5 , U = 0 1 3 4 6 8 0 0 −2 −1 −1 I Then we can write A = L2 L1 U C. Hurtado (UIUC - Economics) Numerical Methods 13 / 39 LU Decomposition LU Decomposition I Notice that: 1 0 0 −1 L1 = 2 1 0 4 0 1 and 1 0 0 −1 L2 = 0 1 0 0 2 1 I The multiplication of lower triangular matrices is also a lower triangular matrix. In particular: 1 0 0 −1 −1 L = L2 L1 = 2 1 0 4 2 1 C. Hurtado (UIUC - Economics) Numerical Methods 14 / 39 LU Decomposition LU Decomposition I In summary: 1 1 1 1 0 0 1 1 1 A = 2 3 5 = 2 1 0 0 1 3 = LU 4 6 8 4 2 1 0 0 −2 I Notice that L is a unit lower triangular matrix, i.e., it has ones on the diagonal. I What should we do for a general matrix A? C. Hurtado (UIUC - Economics) Numerical Methods 15 / 39 LU Decomposition LU Decomposition I Let us start with a11 a12 ··· a1n a21 a22 ··· a2n A = . . .. . an1 an2 ··· ann 1 I Suppose that a11 6= 0. Define li1 = ai1/a11, for i = 2, ··· , n. I Then: 1 1 1 0 0 ··· 0 a11 a12 ··· a1n l1 0 ··· 0 0 a2 ··· a2 21 22 2n (2) L1A = I − . A = . ≡ A . .. . .. . . 1 n n ln1 0 ··· 0 0 an2 ··· ann C. Hurtado (UIUC - Economics) Numerical Methods 16 / 39 LU Decomposition LU Decomposition I Proceeding column by column in similar fashion, we can construct a series of lower triangular matrices that replaces the elements below the diagonal with zeros. k (k) I Let us denote by aij the elements of the k − th matrix A . k I If akk 6= 0, we define ak ij for j = k, i = k + 1, ··· , n k ak lij = kk 0 otherwise and ( ak − lk ak for i = k + 1, ··· , n and j = k, ··· , n ak+1 = ij ik kj ij k aij otherwise C.