18-660: Numerical Methods for Engineering Design and Optimization
Xin Li Department of ECE Carnegie Mellon University Pittsburgh, PA 15213
Slide 1 Overview
Linear Equation Solver LU decomposition Cholesky decomposition
Slide 2 Linear Equation Solver
Gaussian elimination solves a linear equation
A ⋅ X = B
Sometimes we want to repeatedly calculate the solutions for different right-hand-side vectors X B X B A ⋅ 1 = 1 A ⋅ 2 = 2 Case 1 Case 2
Slide 3 Ordinary Differential Equation Example
Backward Euler integration for linear ordinary differential equation with constant time step ∆t
x(t) = A⋅ x(t)+ B ⋅u(t) x(0) = 0
−1 x(tn+1 ) = (I − ∆t ⋅ A) ⋅[x(tn )+ ∆t ⋅ B ⋅u(tn+1 )] x(t0 ) = 0 Identical Different
@ all tn’s @ all tn’s
Slide 4 LU Factorization
It would be expensive to repeatedly run Gaussian elimination for many times How can we save and re-use the intermediate results? LU factorization is to address this problem
X B X B A ⋅ 1 = 1 A ⋅ 2 = 2 Case 1 Case 2
Slide 5 LU Factorization
Key idea: Represent A as the product of L (lower triangular) and U (upper triangular) via Gaussian-elimination-like steps All diagonal elements in U are set to 1 by proper scaling
A ⋅ X = B A = L ⋅ U
LU factorization is unchanged as long as A is unchanged (i.e., independent of the right-hand-side vector B)
Slide 6 LU Factorization
a11 a12 a1N l11 1 u12 u1N a a a l l 1 u 21 22 2N = 21 22 ⋅ 2N aN1 aN 2 aNN lN1 lN 2 lNN 1
li1 ⋅1 = ai1 (i =1,2,, N )
Slide 7 LU Factorization
a11 a12 a1N l11 1 u12 u1N a a a l l 1 u 21 22 2N = 21 22 ⋅ 2N aN1 aN 2 aNN lN1 lN 2 lNN 1
l11 ⋅u1i = a1i (i = 2,3,, N )
Slide 8 LU Factorization
a11 a12 a1N l11 1 u12 u1N a a a l l 1 u 21 22 2N = 21 22 ⋅ 2N aN1 aN 2 aNN lN1 lN 2 lNN 1
li1 ⋅u12 + li2 ⋅1 = ai2 (i = 2,3,, N )
Continue iteration until all elements in L and U are solved
Slide 9 Memory Storage
The matrix A can be iteratively replaced by L and U No additional memory is required
a11 a12 a1N a21 a22 a2N aN1 aN 2 aNN
l11 u12 u1N l21 l22
u − N 1,N lN1 lN 2 lNN
Slide 10 A Simple LU Example
2 1 −1 l11 1 u12 u13 − − = ⋅ 3 1 2 l21 l22 1 u23 − 2 1 2 l31 l32 l33 1
l11 ⋅1 = a11 l21 ⋅1 = a21 l31 ⋅1 = a31
2 1 −1 − 3 −1 2 − 2 1 2
Slide 11 A Simple LU Example
2 1 −1 l11 1 u12 u13 − − ⋅ 3 1 2 l21 l22 1 u23 − 2 1 2 l31 l32 l33 1
l11 ⋅u12 = a12 l11 ⋅u13 = a13
2 1 2 −1 2 − 3 −1 2 − 2 1 2
Slide 12 A Simple LU Example
2 1 2 −1 2 l11 1 u12 u13 − − ⋅ 3 1 2 l21 l22 1 u23 − 2 1 2 l31 l32 l33 1
l21 ⋅u12 + l22 = a22 l31 ⋅u12 + l32 = a32
2 1 2 −1 2 − 3 1 2 2 − 2 2 2
Slide 13 A Simple LU Example
2 1 2 −1 2 l11 1 u12 u13 − ⋅ 3 1 2 2 l21 l22 1 u23 − 2 2 2 l31 l32 l33 1
l21 ⋅u13 + l22 ⋅u23 = a23
2 1 2 −1 2 − 3 1 2 1 − 2 2 2
Slide 14 A Simple LU Example
2 1 2 −1 2 l11 1 u12 u13 − ⋅ 3 1 2 1 l21 l22 1 u23 − 2 2 2 l31 l32 l33 1
l31 ⋅u13 + l32 ⋅u23 + l33 = a33
2 1 2 −1 2 − 3 1 2 1 − 2 2 −1
Slide 15 LU Factorization
Given L and U, solve linear equation via two steps
A⋅ X = B A = LU
L ⋅U ⋅ X = B V
L ⋅V = B U ⋅ X = V
Slide 16 LU Factorization
L ⋅ V = B U ⋅ X = V
Forward substitution Backward substitution
Only the above two steps are repeated if the right-hand-side vector B is changed LU factorization is not repeated More efficient than Gaussian elimination
Slide 17 Cholesky Factorization
If the matrix A is symmetric and positive definite, Cholesky factorization is preferred over LU factorization
A = L ⋅ LT
Cholesky factorization is cheaper than LU Only needs to find a single triangular matrix L (instead of two different matrices L and U)
Slide 18 Cholesky Factorization
A must be symmetric and positive definite to make Cholesky factorization applicable
A symmetric matrix A is positive definite if PT ⋅ A⋅ P > 0 for any real-valued vector P ≠ 0
Sufficient and necessary condition for a symmetric matrix A to be positive definite: All eigenvalues of A are positive
Slide 19 Partial Differential Equation Example
1-D rod discretized into 4 segments ∂ 2T (x,t) κ ⋅ = 0 T1 T2 T3 T4 T5 ∂x2
T1 = 30 T5 =100
κ ⋅(−Ti−1 + 2Ti −Ti+1 ) = 0 (2 ≤ i ≤ 4)
− 30 + 2T2 −T3 = 0
−T2 + 2T3 −T4 = 0
−T3 + 2T4 −100 = 0
Slide 20 Partial Differential Equation Example
− 30 + 2T −T = 0 2 −1 T 30 2 3 2 − + − = − − ⋅ = T2 2T3 T4 0 1 2 1 T3 0 −T3 + 2T4 −100 = 0 −1 2 T4 100
A
Eigenvalues of A
λ1 = 3.41
λ2 = 2.00 (A is positive definite)
λ3 = 0.58
Slide 21 Partial Differential Equation Example
In practice, we never calculate eigenvalues to check if a matrix is positive definite or not Eigenvalue decomposition is much more expensive than solving a linear equation
If we apply finite difference to discretize steady-state heat equation, the resulting linear equation is positive definite
Slide 22 Partial Differential Equation Example
2 −1 l11 l11 l21 l31 − − = ⋅ 1 2 1 l21 l22 l22 l32 −1 2 l31 l32 l33 l33
l11 ⋅l11 = a11 l21 ⋅l11 = a21 l31 ⋅l11 = a31
2 −1 − 1 2 2 −1 0 −1 2
Slide 23 Partial Differential Equation Example
2 −1 l l l l 11 11 21 31 − − ⋅ 1 2 2 1 l21 l22 l22 l32 − 0 1 2 l31 l32 l33 l33
l21 ⋅l21 + l22 ⋅l22 = a22 l21 ⋅l31 + l22 ⋅l32 = a23
2 −1 − 1 2 3 2 −1 0 − 2 3 2
Slide 24 Partial Differential Equation Example
2 −1 l l l l 11 11 21 31 − − ⋅ 1 2 3 2 1 l21 l22 l22 l32 − 0 2 3 2 l31 l32 l33 l33
l31 ⋅l31 + l32 ⋅l32 + l33 ⋅l33 = a33
2 −1 − 1 2 3 2 −1 0 − 2 3 4 3
Slide 25 Summary
Linear equation solver LU decomposition Cholesky decomposition
Slide 26