Part IVa: Gaussian Elimination MA 580; Gaussian Elimination C. T. Kelley NC State University tim [email protected] Version of September 7, 2016 Master Chapters 1--7 of the Matlab book. NOW!!! Read chapter 5 of the notes. Start on Chapter 9 of the Matlab book. NCSU, Fall 2016 Part IVa: Gaussian Elimination c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 1 / 78 Part IVa: Gaussian Elimination Simple Gaussian Elimination Revisited For the second time we solve (I) 2x1 + x2 = 1 (II) x1 + x2 = 2 This time we will record what we do. c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 2 / 78 Part IVa: Gaussian Elimination Pivot element and multiplier We plan to multiply (I) by .5 and subtract from (II). Words: a11 is the pivot element l21 = :5 = a21=a11 is the multiplier I will tell you what L is soon. c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 3 / 78 Part IVa: Gaussian Elimination Solving the system As before: the new (II) is (II)'x2=2 = 1:5 Now I can backsolve the problem starting with x2 = 3. Record the coefficient of x2 in (II)' as u22 = :5. Now plug into (I) to get x1 = −1. In matrix form, there's something profound happening. c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 4 / 78 Part IVa: Gaussian Elimination Encoding the history of Gaussian Elimination Suppose I put the multipliers in a unit lower triangular matrix L 1 0 1 0 L = = l21 1 :5 1 and the coefficients of the backsolve in 2 1 U = : 0 :5 I've encoded all the work in the solve in L and U. In fact 1 0 2 1 2 1 LU = = = A :5 1 0 :5 1 1 c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 5 / 78 Part IVa: Gaussian Elimination The Factor-Solve Paradigm In real life, one separates the factorization A = LU from the solve. First factor A = LU. Then Solve Lz = b (forward solve) Solve Ux = z (back solve) Put this together and Ax = LUx = Lz = b: So you can do several right hand side vectors with a single factorization. c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 6 / 78 Part IVa: Gaussian Elimination Triangular Solves The system for the forward solve looks like 0 1 0 1 0 1 1 0 0 ::: 0 z1 b1 Bl21 1 0 ::: 0 C Bz2 C Bb2 C B C B C B C Bl31 l32 1 0 :::C Bz3 C Bb3 C Lz = B C B C = B C B . .. C B . C B . C @ . 0 A @ . A @ . A lN1 lN2 lN3 ::: 1 zN bN So z1 = b1, z2 = b2 − l21z1, ... c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 7 / 78 Part IVa: Gaussian Elimination Algorithm for Lower Triangular Solve (forward) Forward means solve for zi in order Solve Lz = b with unit triangular L for i = 1 : N do zi = bi for j = 1 : i − 1 do zi = zi − lij zj end for end for i−1 X Bottom line: zi = bi − lij zj ; 1 ≤ i ≤ N: j=1 What if lii 6= 1? c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 8 / 78 Part IVa: Gaussian Elimination Algorithm for Upper Triangular Solve (backward) Backward means solve for zi in reverse order (zN first) Solve Ux = z for i = N : 1 do xi = zi for j = i + 1 : N do xi = xi − uij xj end for xi = xi =uii end for 0 N 1 X Bottom line: xi = @zi − uij xj A =uii ; N ≥ i ≥ 1 j=i+1 c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 9 / 78 Part IVa: Gaussian Elimination Evaluating Cost You can use several equivalent metrics for cost floating point operations floating point multiplications “flops” an add + a multiply + some address computation In MA 580, with one exception, adds and multiplies are paired and the three metrics give equal relative results. c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 10 / 78 Part IVa: Gaussian Elimination Scalar products and matrix multiplies N T X x y = xi yi i=1 N multiplies and N − 1 adds. Cost: N + O(1) leading order is the same for the adds/multiplies Matrix-vector product: N X (Ax)i = aij xj for j = 1 ::: N i=1 N2 multiplies and N2 − N adds (N scalar products) Cost: N2 + O(N) c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 11 / 78 Part IVa: Gaussian Elimination A trick Write the loops as nested sums and add 1 for each multiply. Replace the sums by integrals. Evaluate the integrals You'll be right to leading order. c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 12 / 78 Part IVa: Gaussian Elimination Triangular Solve We'll do lower, upper is the same. Algorithm: for i=1:N do Pi−1 zi = bi − j=1 lij ∗ zj end for N i−1 X X Sum: 1 i=1 j=1 Integral (check it out) Z N Z i−1 dj di = N2=2 + O(N) 1 1 c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 13 / 78 Part IVa: Gaussian Elimination Simple Gaussian Elimination Here are the steps Make a copy of A. Do Gaussian elimination as if one were solving an equation. Recover L and U from our (mangled) copy of A. In the real world, we would overwrite A with the data for the factors, saving storage. Have I told you about the computer scientist who was low on memory? The guy tells me he doesn't have enough bytes. c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 14 / 78 Part IVa: Gaussian Elimination Not enough bytes? So I bit him. H. Youngman c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 15 / 78 Part IVa: Gaussian Elimination Overwriting A with L and U lu simple A A for j = 1 : N do for i = j + 1 : N do aij = aij =ajj fCompute the multipliers. Store them in the strict lower triangle of A.g for k = j + 1 : N do aik = aik − aij ajk fDo the elimination.g end for end for end for c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 16 / 78 Part IVa: Gaussian Elimination Cost of LU factorization N N N X X X Z N Z N Z N 1 ≈ dk di dj = N3=3 + O(N2): j=1 i=j+1 k=j+1 1 j+1 j+1 So, cost of LU is N3=3 + O(N2). c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 17 / 78 Part IVa: Gaussian Elimination Recovering L and U (L; U) recover(A) for j = 1 : N do ljj = 1 for i = 1 : j do uij = aij end for for i = j + 1 : N do lij = aij end for end for c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 18 / 78 Part IVa: Gaussian Elimination Factor-Solve Paradigm: II The factorization costs O(N3) work The upper and lower triangular solves cost O(N2) work So . the standard procedure is to separate them. This makes is much more efficient to handle multiple right hand sides, as we will do in the nonlinear equations part. c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 19 / 78 Part IVa: Gaussian Elimination Mission Accomplished? Not exactly. Does 0 1 A = 1 0 have an LU factorization? No, but solving x b Ax = 2 = b = 1 x1 b2 is pretty easy. We've clearly missed something. c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 20 / 78 Part IVa: Gaussian Elimination Partial pivoting aka column pivoting The easy fix it to interchange the equations (rows) in the problem. Then you get A0 = b0. 1 0 A0 = 0 1 0 T and b = (b2; b1) . Same answer, different problem. Proposed algorithm: When you get a zero pivot, find something below it that is not zero, and swap those rows. Then make sure you don't forget what you did. c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 21 / 78 Part IVa: Gaussian Elimination Are we there yet? How about :1 10 A = 1 0 and b = (20; −1)T . Exact solution: (−1; 2:01)T c C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 22 / 78 Part IVa: Gaussian Elimination Recall the Toy Floating Point Number System Here's a radix 10 system. Two digit mantissa Exponent range: -2:2 Largest float = :99 ∗ 102 = 99 eps(0) = :10 ∗ 10−2 eps(1) = 1:1 − 1 = :1 Round to nearest.