Applications Gaussian Elimination LU Factorization Math 541 - Numerical Analysis Lecture Notes { Linear Algebra: Part A Joseph M. Mahaffy, [email protected] Department of Mathematics and Statistics Dynamical Systems Group Computational Sciences Research Center San Diego State University San Diego, CA 92182-7720 http://jmahaffy.sdsu.edu Spring 2018 Lecture Notes { Linear Algebra: Part A | Joseph M. Mahaffy, [email protected] (1/42) Applications Gaussian Elimination LU Factorization Outline 1 Applications 2 Gaussian Elimination Solving Ax = b Partial Pivoting 3 LU Factorization Example General LU Factorization MatLab Program for solving Ax = b Lecture Notes { Linear Algebra: Part A | Joseph M. Mahaffy, [email protected] (2/42) Applications Gaussian Elimination LU Factorization Applications Applications and Matrices: Widely used in many fields Kirchhoff's Law: Matrices used to find currents in an electric circuit At any node in an electrical circuit, the sum of currents flowing into that node is equal to the sum of currents flowing out of that node The directed sum of the electrical potential differences (voltage) around any closed network is zero Lecture Notes { Linear Algebra: Part A | Joseph M. Mahaffy, [email protected] (3/42) Applications Gaussian Elimination LU Factorization Kirchhoff's Law Kirchhoff's Law applied to the circuit above gives the system of equations 5i1 + 5i2 = V i3 − i4 − i5 = 0 2i4 − 3i5 = 0 i1 − i2 − i3 = 0 5i2 − 7i3 − 2i4 = 0 or AI = B T T with I = [i1; i2; i3; i4; i5] , B = [V; 0; 0; 0; 0] , and 0 5 5 0 0 0 1 B 0 0 1 −1 −1 C B C A = B 0 0 0 2 −3 C B C @ 1 −1 −1 0 0 A 0 5 −7 −2 0 Lecture Notes { Linear Algebra: Part A | Joseph M. Mahaffy, [email protected] (4/42) Applications Gaussian Elimination LU Factorization Kirchhoff's Law { MatLab Solution From above, we want to solve AI = b or 0 1 0 1 0 1 5 5 0 0 0 i1 V B 0 0 1 −1 −1 C B i2 C B 0 C B C B C B C B 0 0 0 2 −3 C B i3 C = B 0 C B C B C B C @ 1 −1 −1 0 0 A @ i4 A @ 0 A 0 5 −7 −2 0 i5 0 If V = 1:5, then MatLab gives the solution 0 1 0 1 i1 0:185047 B i2 C B 0:114953 C B C B C B i3 C = B 0:070093 C B C B C @ i4 A @ 0:042056 A i5 0:028037 Lecture Notes { Linear Algebra: Part A | Joseph M. Mahaffy, [email protected] (5/42) Applications Gaussian Elimination LU Factorization MatLab Solution (Easy) { AI = b There are multiple ways where MatLab solves the above system AI = b Anb inv(A)*b linsolve(A,b) rref([A,b]) Are all of these calculations the same? Which methods are more efficient and why? How does MatLab perform these calculations and what problems arise? Lecture Notes { Linear Algebra: Part A | Joseph M. Mahaffy, [email protected] (6/42) Applications Solving Ax = b Gaussian Elimination Partial Pivoting LU Factorization Linear System Linear System: Operations to simplify E1 : a11x1 + a12x2 + ::: + a1nxn = b1 E2 : a21x1 + a22x2 + ::: + a2nxn = b2 . En : an1x1 + an2x2 + ::: + annxn = bn Ei can be multiplied by a nonzero constant λ with the resulting equation used in place of Ei (Denoted (λEi) ! (Ei).) Ej can be multiplied by any constant λ and added to Ei with the resulting equation used in place of Ei (Denoted (Ei + λEj) ! (Ei).) Ei and Ej can be transposed in order. (Denoted (Ei) ! (Ej).) Lecture Notes { Linear Algebra: Part A | Joseph M. Mahaffy, [email protected] (7/42) Applications Solving Ax = b Gaussian Elimination Partial Pivoting LU Factorization Solving Ax = b Let A be an n × n matrix and x and b be n × 1 vectors. Consider the system of linear equations given by Ax = b The solution set x satisfies one of the following: 1 The system has a single unique solution 2 The system has infinitely many solutions 3 The system has no solution Note that the system has a unique solution if and only if det(A) 6= 0 or equivalently A is nonsingular (it has an inverse) Lecture Notes { Linear Algebra: Part A | Joseph M. Mahaffy, [email protected] (8/42) Applications Solving Ax = b Gaussian Elimination Partial Pivoting LU Factorization Gaussian Elimination Elimination Process: We want to describe the step-by-step process to solve E1 : a11x1 + a12x2 + ::: + a1nxn = a1;n+1 E2 : a21x1 + a22x2 + ::: + a2nxn = a2;n+1 . En : an1x1 + an2x2 + ::: + annxn = an;n+1 Begin by creating the augmented matrix A = [aij] for 1 ≤ i ≤ n and 1 ≤ j ≤ n + 1 We desire a programmable process for creating an equivalent system with an upper triangular matrix, which is then readily solved by backward substitution Lecture Notes { Linear Algebra: Part A | Joseph M. Mahaffy, [email protected] (9/42) Applications Solving Ax = b Gaussian Elimination Partial Pivoting LU Factorization Gaussian Elimination We take the original n × n linear system and create the augmented matrix A = [aij] for 1 ≤ i ≤ n and 1 ≤ j ≤ n + 1 Algorithm (Gaussian Elimination) For i = 1; :::; n − 1, do the next 3 steps 1 Let p be the smallest integer with i ≤ p ≤ n and api 6= 0. If no integer p can be found, then OUTPUT: no unique solution exists and STOP 2 If p 6= i, then perform (Ep) ! (Ei) (pivoting) 3 For j = i + 1; :::; n do the following: 1 Set mji = aji=aii 2 Perform (Ej − mjiEi) ! (Ej ) (producing a leading zero element in Row j) 4 If ann = 0, then OUTPUT: no unique solution exists and STOP Lecture Notes { Linear Algebra: Part A | Joseph M. Mahaffy, [email protected] (10/42) Applications Solving Ax = b Gaussian Elimination Partial Pivoting LU Factorization Back Substitution The previous algorithm produces an augmented matrix with the first n columns creating an upper triangular matrix, U = [uij] Algorithm (Back Substitution) Set xn = un;n+1=unn For i = n − 1; :::; 1 set 2 n 3 1 X x = u − u x i u 4 i;n+1 ij j5 ii j=i+1 OUTPUT (x1; :::; xn) Lecture Notes { Linear Algebra: Part A | Joseph M. Mahaffy, [email protected] (11/42) Applications Solving Ax = b Gaussian Elimination Partial Pivoting LU Factorization Gaussian Elimination Operations The previous algorithms solve Ax = b There were numerous Multiplications/divisions and Additions/subtractions in the Gaussian elimination and back substitution These calculations are readily counted Multiplications/divisions total n3 n + n2 − 3 3 Additions/subtractions total n3 n2 5n + − 3 2 6 which means that arithmetic operations are proportional to n3, the dimension of the system Lecture Notes { Linear Algebra: Part A | Joseph M. Mahaffy, [email protected] (12/42) Applications Solving Ax = b Gaussian Elimination Partial Pivoting LU Factorization Partial Pivoting After pivoting our Algorithm uses the new pivot element to produce 0 below in the remaining rows The operation is mji = aji=aii If aii is small compared to aji, then mji 1, which can introduce significant round-off error Further computations compound the original error In addition, the back substitution using the small aii also introduces more error, which means that the round-off error dominates the calculations Pivoting Strategy: Row exchanges are done to reduce round-off error Lecture Notes { Linear Algebra: Part A | Joseph M. Mahaffy, [email protected] (13/42) Applications Solving Ax = b Gaussian Elimination Partial Pivoting LU Factorization Partial Pivoting { Example Example: Consider the following system of equations: E1 : 0:003000x1 + 59:14x2 = 59:17 E2 : 5:291x1 − 6:130x2 = 46:78 Apply Gaussian elimination to this system with 4-digit arithmetic with rounding and compare to the exact solution, which is x1 = 10:00 and x2 = 1:000 Solution: The first pivot element is a11 = 0:003000, which is small, and its multiplier is 5:291 m = = 1763:666¯ 21 0:003000 rounding to m21 = 1764, which is large Lecture Notes { Linear Algebra: Part A | Joseph M. Mahaffy, [email protected] (14/42) Applications Solving Ax = b Gaussian Elimination Partial Pivoting LU Factorization Partial Pivoting { Example Example (cont): Performing (E2 − m21E1) ! (E2) with appropriate rounding gives 0:003000x1 + 59:14x2 = 59:17 −104300x2 ≈ −104400 while the exact system is 0:003000x1 + 59:14x2 = 59:17 −104309:376¯x2 = −104309:376¯ The disparity in m21a13 and a23 has introduced round-off error, but it has not been propagated Lecture Notes { Linear Algebra: Part A | Joseph M. Mahaffy, [email protected] (15/42) Applications Solving Ax = b Gaussian Elimination Partial Pivoting LU Factorization Partial Pivoting { Example Example (cont): Back substitution yields x2 ≈ 1:001; which is close to the actual value x2 = 1:000 However, the small pivot a11 = 0:003000 gives 59:17 − (59:14)(1:001) x = = −10:00; 1 0:003000 while the actual value is x1 = 10:00 The round-off error comes from the small error of 0.001 multiplied by 59:14 ≈ 20000 0:003000 For this system is is very easy to see where the error occurs and propagates, but it becomes much harder in larger systems Lecture Notes { Linear Algebra: Part A | Joseph M. Mahaffy, [email protected] (16/42) Applications Solving Ax = b Gaussian Elimination Partial Pivoting LU Factorization Partial Pivoting Partial Pivoting: To avoid the difficulty in the previous Example, we select the largest magnitude element in the column below the diagonal and perform a pivoting with this row Specifically, determine the smallest p ≥ k such that japkj = max jaikj k≤i≤n and perform (Ek) ! (Ep) If this is done on the previous Example, then the 4-digit rounding answer agrees with the exact answer Lecture Notes { Linear Algebra: Part A | Joseph M.