18-660: Numerical Methods for Engineering Design and Optimization Xin Li Department of ECE Carnegie Mellon University Pittsburgh, PA 15213 Slide 1 Overview Linear Equation Solver LU decomposition Cholesky decomposition Slide 2 Linear Equation Solver Gaussian elimination solves a linear equation A ⋅ X = B Sometimes we want to repeatedly calculate the solutions for different right-hand-side vectors X B X B A ⋅ 1 = 1 A ⋅ 2 = 2 Case 1 Case 2 Slide 3 Ordinary Differential Equation Example Backward Euler integration for linear ordinary differential equation with constant time step ∆t x(t) = A⋅ x(t)+ B ⋅u(t) x(0) = 0 −1 x(tn+1 ) = (I − ∆t ⋅ A) ⋅[x(tn )+ ∆t ⋅ B ⋅u(tn+1 )] x(t0 ) = 0 Identical Different @ all tn’s @ all tn’s Slide 4 LU Factorization It would be expensive to repeatedly run Gaussian elimination for many times How can we save and re-use the intermediate results? LU factorization is to address this problem X B X B A ⋅ 1 = 1 A ⋅ 2 = 2 Case 1 Case 2 Slide 5 LU Factorization Key idea: Represent A as the product of L (lower triangular) and U (upper triangular) via Gaussian-elimination-like steps All diagonal elements in U are set to 1 by proper scaling A ⋅ X = B A = L ⋅ U LU factorization is unchanged as long as A is unchanged (i.e., independent of the right-hand-side vector B) Slide 6 LU Factorization a11 a12 a1N l11 1 u12 u1N a a a l l 1 u 21 22 2N = 21 22 ⋅ 2N aN1 aN 2 aNN lN1 lN 2 lNN 1 li1 ⋅1 = ai1 (i =1,2,, N ) Slide 7 LU Factorization a11 a12 a1N l11 1 u12 u1N a a a l l 1 u 21 22 2N = 21 22 ⋅ 2N aN1 aN 2 aNN lN1 lN 2 lNN 1 l11 ⋅u1i = a1i (i = 2,3,, N ) Slide 8 LU Factorization a11 a12 a1N l11 1 u12 u1N a a a l l 1 u 21 22 2N = 21 22 ⋅ 2N aN1 aN 2 aNN lN1 lN 2 lNN 1 li1 ⋅u12 + li2 ⋅1 = ai2 (i = 2,3,, N ) Continue iteration until all elements in L and U are solved Slide 9 Memory Storage The matrix A can be iteratively replaced by L and U No additional memory is required a11 a12 a1N a21 a22 a2N aN1 aN 2 aNN l11 u12 u1N l21 l22 u − N 1,N lN1 lN 2 lNN Slide 10 A Simple LU Example 2 1 −1 l11 1 u12 u13 − − = ⋅ 3 1 2 l21 l22 1 u23 − 2 1 2 l31 l32 l33 1 l11 ⋅1 = a11 l21 ⋅1 = a21 l31 ⋅1 = a31 2 1 −1 − 3 −1 2 − 2 1 2 Slide 11 A Simple LU Example 2 1 −1 l11 1 u12 u13 − − ⋅ 3 1 2 l21 l22 1 u23 − 2 1 2 l31 l32 l33 1 l11 ⋅u12 = a12 l11 ⋅u13 = a13 2 1 2 −1 2 − 3 −1 2 − 2 1 2 Slide 12 A Simple LU Example 2 1 2 −1 2 l11 1 u12 u13 − − ⋅ 3 1 2 l21 l22 1 u23 − 2 1 2 l31 l32 l33 1 l21 ⋅u12 + l22 = a22 l31 ⋅u12 + l32 = a32 2 1 2 −1 2 − 3 1 2 2 − 2 2 2 Slide 13 A Simple LU Example 2 1 2 −1 2 l11 1 u12 u13 − ⋅ 3 1 2 2 l21 l22 1 u23 − 2 2 2 l31 l32 l33 1 l21 ⋅u13 + l22 ⋅u23 = a23 2 1 2 −1 2 − 3 1 2 1 − 2 2 2 Slide 14 A Simple LU Example 2 1 2 −1 2 l11 1 u12 u13 − ⋅ 3 1 2 1 l21 l22 1 u23 − 2 2 2 l31 l32 l33 1 l31 ⋅u13 + l32 ⋅u23 + l33 = a33 2 1 2 −1 2 − 3 1 2 1 − 2 2 −1 Slide 15 LU Factorization Given L and U, solve linear equation via two steps A⋅ X = B A = LU L ⋅U ⋅ X = B V L ⋅V = B U ⋅ X = V Slide 16 LU Factorization L ⋅ V = B U ⋅ X = V Forward substitution Backward substitution Only the above two steps are repeated if the right-hand-side vector B is changed LU factorization is not repeated More efficient than Gaussian elimination Slide 17 Cholesky Factorization If the matrix A is symmetric and positive definite, Cholesky factorization is preferred over LU factorization A = L ⋅ LT Cholesky factorization is cheaper than LU Only needs to find a single triangular matrix L (instead of two different matrices L and U) Slide 18 Cholesky Factorization A must be symmetric and positive definite to make Cholesky factorization applicable A symmetric matrix A is positive definite if PT ⋅ A⋅ P > 0 for any real-valued vector P ≠ 0 Sufficient and necessary condition for a symmetric matrix A to be positive definite: All eigenvalues of A are positive Slide 19 Partial Differential Equation Example 1-D rod discretized into 4 segments ∂ 2T (x,t) κ ⋅ = 0 T1 T2 T3 T4 T5 ∂x2 T1 = 30 T5 =100 κ ⋅(−Ti−1 + 2Ti −Ti+1 ) = 0 (2 ≤ i ≤ 4) − 30 + 2T2 −T3 = 0 −T2 + 2T3 −T4 = 0 −T3 + 2T4 −100 = 0 Slide 20 Partial Differential Equation Example − 30 + 2T −T = 0 2 −1 T 30 2 3 2 − + − = − − ⋅ = T2 2T3 T4 0 1 2 1 T3 0 −T3 + 2T4 −100 = 0 −1 2 T4 100 A Eigenvalues of A λ1 = 3.41 λ2 = 2.00 (A is positive definite) λ3 = 0.58 Slide 21 Partial Differential Equation Example In practice, we never calculate eigenvalues to check if a matrix is positive definite or not Eigenvalue decomposition is much more expensive than solving a linear equation If we apply finite difference to discretize steady-state heat equation, the resulting linear equation is positive definite Slide 22 Partial Differential Equation Example 2 −1 l11 l11 l21 l31 − − = ⋅ 1 2 1 l21 l22 l22 l32 −1 2 l31 l32 l33 l33 l11 ⋅l11 = a11 l21 ⋅l11 = a21 l31 ⋅l11 = a31 2 −1 − 1 2 2 −1 0 −1 2 Slide 23 Partial Differential Equation Example 2 −1 l l l l 11 11 21 31 − − ⋅ 1 2 2 1 l21 l22 l22 l32 − 0 1 2 l31 l32 l33 l33 l21 ⋅l21 + l22 ⋅l22 = a22 l21 ⋅l31 + l22 ⋅l32 = a23 2 −1 − 1 2 3 2 −1 0 − 2 3 2 Slide 24 Partial Differential Equation Example 2 −1 l l l l 11 11 21 31 − − ⋅ 1 2 3 2 1 l21 l22 l22 l32 − 0 2 3 2 l31 l32 l33 l33 l31 ⋅l31 + l32 ⋅l32 + l33 ⋅l33 = a33 2 −1 − 1 2 3 2 −1 0 − 2 3 4 3 Slide 25 Summary Linear equation solver LU decomposition Cholesky decomposition Slide 26 .