Chapter 3
Matrix Computation
The main objective of this chapter is to solve the linear system
Ax = f, where A is a square n n matrix, and x and f are vectors of order n. × 3.1 Basics Matrix. a a a 11 12 · · · 1n a21 a22 a2n A = . . · · · . = [aij ] i, j = 1, 2, ..., n. . . . a a a n1 n2 · · · nn Column Vectors.
x1 f1 x2 f2 x = . , f = . . . x f n n Row Vectors.
T x = (x1, x2, . . . , xn)
T f = (f1, f2, . . . , fn)
Inner Products.
b1 n b2 T a b = (a1, a2, . . . , an) . = aibi scalar . ← i=1 b ' n Operation count: n multiplications, n 1 additions 2n operations. − $
29 Chap. 3. Matrix Computation CS414 Class Notes 30
Outer Products. a a b a b a b 1 1 1 1 2 · · · 1 n a2 a2b1 a2b2 a2bn T ab = . (b1, b2, . . . , bn) = . . · · · . matrix . . . . ← a a b a b a b n n 1 n 2 · · · n n Operation count: n2 multiplications = n2 operations.
3.1.1 Setting up matrix problems in MATLAB Hilbert matrix. Suppose we want to construct a matrix A s.t. 1 A = ij i + j 1 − MATLAB
A = zeros(n,n); for i = 1:n, for j = 1:n, A(i,j) = 1/(i+j-1); end; end;
This is known as a Hilbert matrix, and can also be created in MATLAB by A = hilb(n). For n = 5, this matrix is of the form 1 1/2 1/3 1/4 1/5 1/2 1/3 1/4 1/5 1/6 A = 1/3 1/4 1/5 1/6 1/7 1/4 1/5 1/6 1/7 1/8 1/5 1/6 1/7 1/8 1/9 Observing that the Hilbert matrix is symmetric, the above MATLABcode fragment could be modified to take advantage of this symmetry: MATLAB A = zeros(n,n); for i = 1:n, for j = i:n, A(i,j) = 1/(i+j-1); A(j,i) = A(i,j); end; end;
Another matrix. As another example of matrix set-up, consider the following MATLAB code: MATLAB P = zeros(n,n); P(:,1) = ones(n,1); % P(:,1) = 1st column of P for i = 2:n, for j = 2:i, P(i,j) = P(i-1,j-1)+P(i-1,j); end; end; Chap. 3. Matrix Computation CS414 Class Notes 31
For n = 5, the above code generates the following matrix 1 0 0 0 0 1 1 0 0 0 P = 1 2 1 0 0 1 3 3 1 0 1 4 6 4 1 Next, we present two special matrices that can be created given a vector of n components, i.e., the elements of the matrix depend only on n parameters.
Vandermonde matrix. This a special type of matrix that can be created from a vector of n components. Given a vector x1 x2 x = . . x n we can create a matrix V as follows: MATLAB n = length(x); V(:,1) = ones(n,1); for j = 2:n, V(:,j) = x .* V(:,j-1); end; Over here, the jth column of V is obtained by element-wise multiplication of the vector x with the (j 1)th column. For n = 4, V is of the form − 2 3 1 x1 x1 x1 2 3 1 x2 x2 x2 V = 2 3 1 x3 x3 x3 2 3 1 x4 x x 4 4 Circulant matrix. Another matrix that can be created from a vector of size n is the circulant matrix. Given a vector a1 a2 a = . . a n we can create a circulant matrix C as follows: MATLAB function C = circulant(a) n = length(a,1); C(1,:) = a’; for j = 2:n, C(i,:)=[C(i-1,n) C(i-1,1:n-1)]; end; Over here, the ith row of C is obtained by permuting (i 1)th row; the last element of (i 1)th row is permuted to the start of the ith row. For n = 4, C is of the− form −
a1 a2 a3 a4 a a a a C = 4 1 2 3 a3 a4 a1 a2 a2 a3 a4 a1 Chap. 3. Matrix Computation CS414 Class Notes 32
3.1.2 Structure of Matrices The nonzero elements of a matrix determine its structure. Following are commonly occurring matrices. ! 0 0 0 0 ! ! 0 0 0 0 ! 0 0 0 ! ! ! 0 0 0 0 ! 0 0 0 ! ! ! 0 0 0 0 ! 0 0 0 ! ! ! 0 0 0 0 ! 0 0 0 ! ! Diagonal Tridiagonal
! 0 0 0 0 ! ! ! ! ! ! ! ! ! ! ! ! 0 0 0 0 ! ! ! ! ! ! ! ! ! ! ! ! 0 0 0 0 ! ! ! 0 ! ! ! ! ! ! ! ! 0 0 0 0 ! ! 0 0 ! ! ! ! ! ! ! ! 0 0 0 0 ! 0 0 0 ! ! Lower triangular Upper triangular Upper Hessenberg A diagonal matrix can be constructed from a vector d, where d = [d d d d ]; 1 2 3 · · · n by the MATLAB statement D = diag(d). This constructs the diagonal matrix D, s.t.,
d1 d2 D = . .. d n MATLAB also provides a way to specify diagonal entries other than the main diagonal. For example, consider the following code: MATLAB m = 3; k = 2; v = [10 20 30]; A = diag(v,k); This constructs a matrix A with the vector v forming the kth diagonal of A: 0 0 10 0 0 0 0 0 20 0 A = 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 Also, the statement v = diag(A,k) extracts the kth diagonal in the form of a vector. A negative k refers to diagonals below the main diagonal while k = 0 refers to the main diagonal.
Block structures. Matrices with block structure can also be constructed in MATLAB. A block structured matrix consists of elements that are in turn smaller matrices. For example, a block structured matrix of the form A11 A12 (3 3) (3 2) × × A = A21 A22 (2 3) (2 2) × × can be constructed from the following code MATLAB Chap. 3. Matrix Computation CS414 Class Notes 33
A11 = rand(3,3); A12 = rand(3,2); A21 = [1 2 3 ; 4 5 6]; A22 = [7 8 ; 9 10]; A = [A11 A12 ; A21 A22];
3.2 Matrix Operations
Matrix-Vector Multiplication. Given an m n matrix A and a vector x of size n, we wish to compute the vector y of size m s.t. × y = Ax where a a a x 11 12 · · · 1n 1 a21 a22 a2n x2 A = . . · · · x = . . . . a a a x m1 m2 · · · mn n gives the vector y with elements n yi = aikxk i = 1, 2, . . . , m 'k=1 Operation count: mn multiplications, m(n 1) additions 2mn operations. − $ MATLAB y = zeros(m,1); for i = 1:m, for j = 1:n, y(i) = y(i) + A(i,j) * x(j) end; end; This can be computed in a number of ways. Alternative 1: (row-oriented)
T T a1 x1 a1 x T T a2 x2 a2 x A = . , x = . y = . . . ⇒ . aT x aT x m n m The corresponding MATLAB program uses m inner products: MATLAB for i = 1:m, y(i) = A(i,:) * x; end; Alternative 2: (column-oriented)
x1 n x2 A = (b1, b2, . . . , bn), x = . y = xjbj . ⇒ j=1 x ' n The corresponding MATLAB program uses summation of n vectors, i.e., “scalar * vector + vector”: MATLAB Chap. 3. Matrix Computation CS414 Class Notes 34
for j = 1:n, y = y + x(j) * A(:,j); end;
Matrix-Matrix Multiplication. Given an m r matrix A and an r n matrix B we wish to compute the product C of size m n s.t. × × × C = A B ∗ (m ↓ n) (m ↓ r) (r ↓ n) × × × The matrix C can be represented as
i = 1, 2, . . . , m C = [c ], ij j = 1, 2, . . . , n where r cij = aikbkj k'=1 Operation count: mnr multiplications, mn(r 1) additions 2mnr operations. − $ MATLAB C = zeros(m,n); for j = 1:n, for i = 1:m, for k = 1:r, C(i,j) = C(i,j) + A(i,k)*B(k,j); end; end; end;
Alternative 1: (inner products)
T a1 aT 2 T A = . , B = (b1, b2, . . . , bn) cij = ai bj . ⇒ aT m MATLAB for j = 1:n, for i = 1:m, C(i,j) = A(i,:)*B(:,j); end; end;
Alternative 2: (outer products)
T h1 hT r 2 T A = (g1, g2, . . . , gr) B = . , C = gkhk m n matrix . ⇒ ← × k=1 hT ' r MATLAB Chap. 3. Matrix Computation CS414 Class Notes 35
for k = 1:r, C = C + A(:,k)*B(k,:); end;
Alternative 3: (matrix-vector products)
B = (b , b , . . . , b ) C = (Ab , Ab , . . . , Ab ) 1 2 n ⇒ 1 2 n
MATLAB for j = 1:n, C(:,j) = A*B(:,j); end;
3.2.1 Matrix Norm, Inverse, and Sensitivity of Linear Systems Inverse of a Matrix. If A is a square matrix of order n, then A is nonsingular if there exists a unique n n matrix X such that × 1 1 AX = XA = I = . .. 1 1 The inverse, X, is also denoted by A− . The following property holds for inverse of a product of two matrices:
1 1 1 (A B)− = B− A− . ∗ ∗ Note also that (A B)T = BT AT . ∗ ∗ The identity matrix I is often expressed as
T I = (e1, e2, . . . , en), ei = (0, 0, . . . , 0, 1, 0, . . . , 0), i.e., ei has a one in the ith position and zero everywhere else.
Vector and matrix norms. Suppose that x" and x"" are two approximations of x. Let
1 1.01 1 x = 1 , x" = 1.01 , x"" = 1.02 . 1 1.01 1 Based on the errors in the approximation, can we decide which one is a better approximation of x?
0.01 0.0 ∆x" = x" x = 0.01 , ∆x"" = x"" x = 0.02 . − 0.01 − 0.0 A norm is a mapping that assigns to each vector a nonnegative real number, i.e., it is a generalization of the scalar absolute value. We define the maximum norm or the infinity norm of a vector as the largest absolute value of all the elements, i.e.,
x = max xi = x ( ( 1 i n | | ( (∞ ≤ ≤ In MATLAB this is computed by norm(x,inf). In our example, ∆x" = 0.01 ∆x"" = 0.02. ( ( ( ( Chap. 3. Matrix Computation CS414 Class Notes 36
Thus, using the max-norm, we conclude that x" is a better approximation of x. (The outcome could be different if we use a different norm.) Norms can be defined for matrices in a similar way. The matrix norm corresponding to the vector max-norm is given by, n A = max aij = A ( ( 1 i n | | ( (∞ ≤ ≤ j=1 ' Clearly, this equals the maximum row sum of the absolute values of the matrix. In MATLAB this is computed by norm(A,inf).
Example 3.1 1 2 A = − A = max 3, 1 = 3 0 1 ⇒ ( ( { } ( ) 2 A number of norms can be defined. The most common of these are presented below:
n 1. x = x . ( (1 | i| i=1 ' T 1 2. x = (x x) 2 . ( (2 n 3. A 1 = max aij (max. col. sum). ( ( 1 j n | | ≤ ≤ i=1 ' 1 2 4. A = a 2 (Frobenius). ( (F | ij | j i ' ' 5. A 2 = max Ax 2 (spectral). ( ( x 2=1( ( % %
Properties of norms. 1. x > 0; moreover, x = 0 if and only if x = 0. ( ( ( ( 2. αx = α x . ( ( | |( ( 3. x + y x + y . ( ( ≤ ( ( ( ( 4. A > 0; moreover, A = 0 if and only if A = 0. ( ( ( ( 5. αA = α A . ( ( | |( ( 6. Ax A x . ( ( ≤ ( (( ( 7. A + B A + B . ( ( ≤ ( ( ( (
Theorem 3.1 If Aˆ is the stored version of the matrix A on a computer with # unit roundoff, then Aˆ = A+E where E # A . ( ( ≤ ∗ ( ( Proof We know that Aˆ = [aˆij ], aˆij = fl(aij ) = aij (1 + #ij ) Chap. 3. Matrix Computation CS414 Class Notes 37 with # # (unit roundoff). Now, | ij | ≤ E = Aˆ A ( ( ( − n( = max aˆij aij 1 i n | − | ≤ ≤ j=1 ' n max aij #ij ≤ 1 i n | | ≤ ≤ j=1 ' n # max aij ≤ ∗ 1 i n | | ≤ ≤ j=1 ' = # A . ∗ ( ( 2
Residual and Error Vectors. Consider the linear system Ax = f. Suppose that the computed solution is xˆ. Then we define the following: residual: r = f Axˆ • − error: δx = xˆ x • − Therefore, r = f Axˆ = Ax Axˆ = A(x xˆ) = Aδx − − − − In some situations one requires a small residual, i.e., r tolerance, and the size of the error is of no consequence. In other situations, however, one needs to(obtain( ≤ small errors, i.e., δx small quantity. For example, consider the linear system ( ( ≤
0.780 0.563 x 0.217 1 = . 0.913 0.659 x 0.254 ( ) ( 2 ) ( ) The solution is given by 1 x = . 1 ( − ) Consider an approximate solution xˆa
6 0.341 10− 0.659 xˆa = , ra = , δxa = xˆa x = − . 0.087 0 − 0.913 ( − ) ( ) ( ) 6 The residual and error norms are r = 10− and δx 0.9, respectively. On the other hand, for another ( a( ( a( $ computed solution xˆb,
0.999 0.000780 0.001 xˆb = , rb = , δxb = xˆb x = − , 1.000 0.000913 − 0 ( − ) ( ) ( ) Clearly, in the first case, the solution is meaningless in spite of the smaller residual norm!
Problem Sensitivity. In the preceding 2 2 system, the matrix × 0.780 0.563 A = 0.913 0.659 ( ) 6 has a determinant, det(A) 10− . In fact $ 0.780 0.563001095 . . . A˜ = 0.913 0.659 ( ) Chap. 3. Matrix Computation CS414 Class Notes 38
6 is exactly singular; i.e., a perturbation of roughly 10− of the element in the (1,2) position renders the problem insoluble. In other words, the above problem is ill-conditioned. Consider the hypothetical situation in which there is no roundoff error during the entire solution process except when A and f are stored. Thus, we have the system
(A + ∆A)(x + ∆x) = (f + ∆f).
It can be shown that ∆x ∆A ∆f ( ( β ( ( + ( ( x ≤ ∗ A f ( ( . ( ( ( ( / where the “magnification factor” β is given by
κ(A) β = ∆A 1 κ(A) % A % − % % in which ∆A κ(A)( ( < 1. A ( ( Here, 1 κ(A) = A A− ( (( ( is called the condition number of A with respect to the max. norm. Note that κ(A) 1. To see that, recall 1 ≥ that AA− = I, and 1 1 I = 1 = AA− A A− . ( ( ( ( ≤ ( (( ( Moreover, if ∆A ∆f ( ( #, ( ( #, [# = unit roundoff] A ≤ f ≤ ( ( ( ( then ∆x ( ( < κ(A) #, x ∗ ( ( ∼ ∆x i.e., % % is less than a quantity that is proportional to κ(A) #. Furthermore, if κ(A) # is close to 1 then x ∗ ∗ A is “n% umerically”% singular.
Example 3.2 Consider the linear system Ax = f in which the representation error of A is given by
∆A 16 ( ( 10− . A $ ( ( If κ(A) 1011, give an upper bound on the rel. error in the solution ∆x / x . $ ( ( ( ( Solution 11 ∆x κ(A) ∆A 10 16 5 ( ( ∆A ( ( 5 (10− ) 10− . x ≤ A $ 1 10− $ 1 κ(A) % A % . / ( ( − % % ( ( − 2 Example 3.3 Let 4.1 2.8 1 66 28 A = . A− = . 9.7 6.6 −97 41 ( ) ( − ) 1 Verify that AA− = I. Assuming that the relative perturbation in the right-hand side is ∆f ( ( 0.000613, f $ ( ( obtain an upper bound for the rel. error in the solution ∆x / x . ( ( ( ( Chap. 3. Matrix Computation CS414 Class Notes 39
Solution We have A = max 6.9, 16.3 = 16.3 ( ( 1 { } A− = max 94, 138 = 138 (κ(A) ( = (16.3)(138){ } 2249.4 $ Now, ∆x ∆f ( ( κ(A) ( ( 1.38 x ≤ f $ ( ( . ( ( / In fact, if 1 0.06 0.94 x = , xˆ = , i.e., ∆x = , 1 2.38 −1.38 ( ) ( ) ( ) then ∆x ( ( = 1.38 x ( ( exactly! 2
3.3 Solution of Linear Systems 3.3.1 Gaussian Elimination Consider the system of equations:
3x1 + 6x2 + 9x3 = 39 Eq.(1) 2x2 + 5x2 2x3 = 3 Eq.(2) x + 3x − x = 2 Eq.(3) 1 2 − 3 These equations can be solved by the process of Gaussian Elimination. Gaussian Elimination consists of two stages - forward elimination and back substitution.
Stage I. Forward elimination. The goal of this stage is to use the equations numbered (1)–(k) to eliminate the terms with unknowns x1, x2, . . . , xk from equations numbered (k+1)–(n). This is achieved in n 1 steps. For our example system, the steps are as follows. − 2 1 Step 1: Eliminate x1 from Eq.(2) by adding 3 Eq.(1) to Eq.(2), and from Eq.(3) by adding 3 Eq.(1) to Eq.(3). The linear system becomes: − ∗ − ∗
3x1 + 6x2 + 9x3 = 39 Eq.(1) 0 + x2 8x3 = 23 Eq.(2) 0 + x −+ 4x = −11 Eq.(3) 2 3 − Note that the right-hand side of Eq.(2) and Eq.(3) are also updated similarly.
Step 2: Eliminate x2 from Eq.(3) by subtracting Eq.(2) from Eq.(3). Now, the linear system is
3x1 + 6x2 + 9x3 = 39 Eq.(1) x 8x = 23 Eq.(2) 2 − 3 − 4x3 = 12 Eq.(3)
At the end of forward elimination stage, Eq. (k) has terms with unknowns xk, . . . , xn only, i.e., the linear system consists of an upper triangular matrix. The next stage computes the values of unknowns in reverse order, i.e., xn, xn 1, . . . , x1, by using equations in that same order. − Chap. 3. Matrix Computation CS414 Class Notes 40
Stage II. Back substitution. The linear system can be rewritten as follows:
3x1 + 6x2 + 9x3 = 39 Eq.(1) x 8x = 23 Eq.(2) 2 − 3 − 4x3 = 12 Eq.(3)
Step 1: Obtain x3 from Eq. (3), i.e., x3 = 3. Step 2: Obtain x from Eq. (2), i.e., x = 8x 23 = 1. 2 2 3 − Step 3: Substitute x2 and x3 in Eq. (1) to get x1, i.e.,
3x = 39 6x 9x = 6 x = 2. 1 − 2 − 3 ⇒ 1 The solution obtained at the conclusion of the algorithm is
2 x = 1 . 3 Gaussian Elimination in Matrix Notation. Our example can be written in matrix form:
3 6 9 x1 39 2 5 2 x2 = 3 , 1 3 −1 x 2 − 3 where 3 6 9 39 A = 2 5 2 f = 3 . 1 3 −1 2 − The operations in the forward elimination stage canalso be represen ted by matrices multiplying A. Let Ai 1x = fi 1 be the linear system at the start of step i, and let Mi be the matrix that is multiplied to Ai 1 − − − to obtain Ai.
Stage I. Forward elimination.
Step 1: At the start of this step, we have the linear system A0x = f0, where A0 = A and f0 = f. The transformation of the linear system at this step is given by M1A0x = M1f0. It is easy to see that
1 3 6 9 x1 1 39 2 2 3 1 2 5 2 x2 = 3 1 3 , − 1 1 1 3 −1 x − 1 1 2 − 3 − 3 − 3 3 6 9 x1 39 0 1 8 x2 = 23 ⇒ 0 1 −4 x −11 − 3 − and thus, 1 2 M1 = 3 1 − 1 1 − 3 Step 2: The next transformation is given by M2A1x = f1, where A1 and f1 are obtained in the previous step. Observe that
1 3 6 9 x1 1 39 1 1 8 x2 = 1 23 1 1 1 −4 x 1 1 −11 − − 3 − − Chap. 3. Matrix Computation CS414 Class Notes 41
3 6 9 x1 39 1 8 x = 23 ⇒ − 2 − 4 x3 12 and thus, 1 M2 = 1 1 1 − The linear system at the end of this step is A2x = f2. The matrix A2 is an upper triangular matrix, and the corresponding system is solved by back substitution.
Gaussian Elimination for General n. Let us develop the Gaussian Elimination algorithm to solve the linear system Ax = f where A is a nonsingular matrix of size n n and x and f are vectors of size n. The forward elimination stage consists of n 1 steps to convert A into× an upper triangular matrix. − Forward elimination
A0 = A, f0 = f for k = 1, 2, . . . , n 1 − MkAk 1x = Mkfk 1 − − [Ak = MkAk 1, fk = Mkfk 1] end; − −
Back-substitution is used to solve the upper triangular system obtained from the forward elimination stage: An 1x = fn 1. −Let us no−w compute the matrices M , k = 1, 2, . . . , n 1. Suppose k − (0) (0) (0) a11 a12 a1n (0) (0) · · · (0) a21 a22 a2n (0) (0) · · · (0) A = A = a a a . 0 31 32 · · · 3n . . . . . . (0) (0) (0) a a ann n1 n2 · · · (k) where the symbol aij is used to denote the value of the element aij at the start of the kth step.
Forward elimination step 1. The matrix M1 that transforms all the elements below a11 in the first column of A0 to zero is given by 1 m21 1 −m 1 M1 = − 31 . . . .. m 1 − n1 where (0) ai1 mi1 = (0) , i = 2, . . . , n a11 The transformed matrix A1 = M1A0 is given by (0) (0) 0) (0) a11 a12 a13 a1n (1) (1) · · · (1) 0 a22 a23 a2n (1) (1) · · · (1) A = 0 a a a 1 32 33 · · · 3n . . . . . . . . 0 a(1) a(1) a(1) n2 n3 nn · · · Chap. 3. Matrix Computation CS414 Class Notes 42 where i = 2, . . . , n a(1) = a(0) m a(0) , ij ij − i1 1j j = 1, . . . , n It is important to note that the first row remains unchanged.
Forward elimination step 2. At this step, M2 is used to transforms all the elements below a22 in the second column of A1 to zero. This is given by 1 1 m 1 − 32 M2 = m42 1 , − . . . .. m 1 − n2 where (1) ai2 mi2 = (1) , i = 3, . . . , n a22
The transformed matrix A2 = M2A1 is given by
(0) (0) (0) (0) a11 a12 a13 a1n (1) (1) · · · (1) 0 a22 a23 a2n (2) · · · (2) A = 0 0 a a 2 33 · · · 3n . . . . . . . . 0 0 a(2) a(2) n3 nn · · · where i = 3, . . . , n a(2) = a(1) m a(1) , ij ij − i2 2j j = 2, . . . , n Observe that at this step, the first and second rows remain unchanged.
Forward elimination step k. At step k, the matrix Ak 1 has the following structure − a(0) a(0) a(0) a(0) a(0) 11 12 13 · · · 1k · · · 1n a(1) a(1) a(1) a(1) 22 23 · · · 2k · · · 2n a(2) a(2) a(2) 33 · · · 3k · · · 3n .. . . Ak 1 = . . . − (k 1) (k 1) 0 akk− akn− · · · . . . . (k 1) (k 1) a − ann− nk · · · and the matrix Mk used to transform all the elements below akk in the kth column of Ak 1 to zero is given by − 1 .. . 1 M = mk+1,k 1 , k − mk+2,k 1 − . . . .. m 1 − nk Chap. 3. Matrix Computation CS414 Class Notes 43 where (k 1) aik− mik = (k 1) , i = k + 1, . . . , n akk−
The elements of the transformed matrix Ak = MkAk 1 are − (k 1) (k 1) aij − mikakj− , i, j = k + 1, . . . , n (k) − aij = 0, i k + 1, j k (k 1) ≥ ≤ a − , i k ij ≤
The reader is encouraged to verify that the kth column below diagonal in Ak is zero. At the kth step, the right-hand side is also transformed by Mk, and the resulting vector fk = Mkfk 1 is given as − (k 1) (k 1) (k) fi − mikfk − , i = k + 1, . . . , n fi = (k 1) − f − , i k 4 i ≤ (k 1) (k 1) The forward elimination algorithm has the potential to break down when akk− = 0 or akk− 0. This situation can be rectified by using pivoting techniques; this will be discussed in later sections. ≈
Back substitution. At this stage, we need to solve the upper triangular system Ux = g, where
(0) (0) (0) (0) a11 a12 a1n f1 (1) · · · (1) (1) a22 a2n f2 U = An 1 = · · · . g = fn 1 = . − .. . − . . . . (n 1) (n 1) a − f − nn n Gaussian Elimination in MATLAB. We now present a MATLAB program for the forward elimination and back substitution stages of Gaussian Elimination. MATLAB % FORWARD ELIMINATION m = zeros(n,1); for k=1:n-1, % Compute kth column of M{k} m(k+1:n) = A(k+1:n,k)/A(k,k); % Multiply M{k} with A{k-1} for i=k+1:n, A(i,k+1:n)=A(i,k+1:n)-m(i)*A(k,k+1:n); end; % Update f{k} f(k+1:n) = f(k+1:n) - m(k+1:n) * f(k); end; U = triu(A); % Extract upper triangular part
% BACK SUBSTITUTION x(n) = f(n)/U(n,n); for k=n-1:-1:1, f(1:k) = f(1:k)-U(1:k,k+1)*f(k+1); f(k) = f(k)/U(k,k); end;
The operation count for the forward elimination stage is given below: Chap. 3. Matrix Computation CS414 Class Notes 44
Matrix Operations: n 1 − 1 Div: (n k) = n(n 1) (compute M ) − 2 − k k=1 n'1 − 1 Mult: (n k)2 = n(n 1)(2n 1) (compute A ) − 6 − − k k=1 n'1 − 1 Add: (n k)2 = n(n 1)(2n 1) (compute A ) − 6 − − k k=1 Total: T'= 2 n3 1 n2 1 n u 3 − 2 − 6 Right-Hand Side: n 1 − 1 Mult: (n k) = n(n 1) − 2 − k=1 n'1 − 1 Add: (n k) = n(n 1) − 2 − k=1 Total: T'= n(n 1) f − The operation count for the back substitution stage is given below: Div: n n 1 − 1 Mult: (n k) = n(n 1) − 2 − k=1 n'1 − 1 Add: (n k) = n(n 1) − 2 − k=1 ' 2 Total: Tb = n Recall the formulas for summation: m m 1 1 j = m(m + 1), j2 = m(m + 1)(2m + 1) 2 6 j=1 j=1 ' ' Thus, the overall time for the solution of a linear system of order n using Gaussian Elimination is
2 3 3 2 7 T ∗ = T + T + T = n + n n u f b 3 2 − 6 In general, we say that Gaussian Elimination takes O(n3) operations. • Back-substitution takes O(n2) operations. • 3.3.2 Triangular Factorization: LU Decomposition Let us again consider the system Ax = f. The forward elimination process may be written as:
Mn 1Mn 1 M2M1A = An 1 = U. − − · · · − 1 1 1 Multiplying both sides by the matrix M1− M2− Mn− 1, we have · · · − 1 1 1 A = M1− M2− Mn− 1U · · · − Recall that each Mk is a lower triangular matrix. In fact, Mk is a unit lower triangular matrix which is a 1 lower triangular matrix with 1’s on diagonal. Furthermore, each Mk− is unit lower triangular, and so is the 1 1 product matrix M1− Mn− 1. Let L be the unit lower triangular matrix defined as · · · − 1 1 1 L = M1− M2− Mn− 1, · · · − Chap. 3. Matrix Computation CS414 Class Notes 45 then we have A = LU. With this result, we have expressed a general nonsingular matrix A as a product of a unit lower and an upper triangular matrix. The result would be much more useful if we could construct L easily. We now describe how to obtain L from the forward elimination stage. The following assertions are easy to verify. It can be shown that • 1 .. . 1 1 M − = mk+1,k 1 , k mk+2,k 1 . . . .. m 1 nk 1 i.e., Mk− is identical to Mk with the exception that the sign on mik is reversed. It can be also be shown that • 1 .. . 1 1 1 mk+1,k 1 Mk− Mk−+1 = . m m .. k+2,k k+2,k+1 . . . . m m 1 nk nk+1 Finally, using the previous result, it can be shown that • 1 m21 1 m31 m32 1 L = . . . . . . .. . . . . 1 mn1 mn2 mn,n 1 1 − Solution of Linear Systems Using LU Decomposition. The factorization of a matrix A into the triangular factors L and U can be used to solve the linear system Ax = f. Since A = LU, we need to solve the system LUx = f. Assuming y = Ux, the solution consists of the following two steps. 1. Solve Ly = f. 2. Solve Ux = y. The reader can verify that the first step corresponds to the transformation of f during the forward elimination stage, and that indeed y = fn 1. The operation count for LU− decomposition of a matrix is 2 T < n3 u 3 Chap. 3. Matrix Computation CS414 Class Notes 46 and for the solution of the triangular systems with matrices L and U is
2 Ts < 2n .
The advantage of computing LU decomposition over Gaussian Elimination is evident when solving a system with multiple right-hand sides. For example, when solving AX = F , where F is an n m matrix, the operation count is × 2 3 2 T ∗ = T + T n + 2mn LU u s ≈ 3 whereas using Gaussian Elimination for each right-hand side, the operation count would have been
2 3 T ∗ = mn , GE 3 which is a factor of m more! Example 3.4 Compute the LU decomposition of A from our earlier example:
3 6 9 A = 2 5 2 1 3 −1 − We know that 1 1 3 6 9 2 M1 = 3 1 , M2 = 1 , U = 0 1 8 − 1 1 1 1 0 0 −4 − 3 − Thus, 1 1 1 1 1 2 2 L = M1− M2− = 3 1 1 = 3 1 1 1 3 1 1 1 3 1 1 2
3.3.3 Pivoting for Stability (k 1) Gaussian elimination, as described above, will break down if akk− = 0 for any k = 1, . . . , n 1. the following example illustrates this for a 2 2 linear system. − × Example 3.5 Consider the system 0 x1 + 1 x2 = 1 1 ∗ x + 1 ∗ x = 2 ∗ 1 ∗ 2 or 0 1 x 1 1 = 1 1 x 2 ( ) ( 2 ) ( ) with the exact solution: 1 x = . 1 ( ) (0) Clearly, Gaussian elimination fails in the first step since a11 = 0. If we exchange the rows, however, we get
1 1 x 2 1 = 0 1 x 1 ( ) ( 2 ) ( ) and proceed with back substitution. Row 2, which is now row 1, is called the “pivotal row”. 2 An important concept in numerical computations is the following: Chap. 3. Matrix Computation CS414 Class Notes 47
If a mathematical process fails when a particular parameter achieves a certain value, then the corresponding numerical algorithm will most likely be unstable when his parameter is near the critical value. A slight perturbation in the linear system from the previous example serves to illustrate this point.
Example 3.6 Consider the linear system Ax = f where
10 4 1 1 A = − f = . 1 1 2 ( ) ( ) Solve the above linear system with Gaussian elimination, using 3-digit decimal arithmetic with rounding. Solution The first step in forward elimination gives
1 0 m = 1/10 4 = 104, M = 21 − 1 104 1 ( − ) and 10 4 1 1 fl(M A) = − , fl(M f) = . 1 0 fl(1 104) 1 fl(2 104) ( − ) ( − ) But, fl(1 104) = fl[105( 0.100000) + 101(0.100000)] − = fl[105(−0.100000 + .000010)] = fl[105(−.099990)] = 105(0−.100) = 104, fl(2 104) = −104. − − − Thus, we have 10 4 1 x 1 − 1 = 0 104 x 104 ( − ) ( 2 ) ( − ) and the approximate solution is 0 xˆ = . 1 ( ) The exact solution, however, is given by
1.000100010001 x = . 0.999899989998 ( ) 4 Now, if we exchange rows so that m < 1, i.e., m = 10− , then | 21| 21 1 1 1 1 2 fl(A1) = 4 = fl(f1) = , 0 fl(1 10− ) 0 1 1 ( − ) ( ) ( ) and the solution is 1 xˆ = , 1 ( ) which is a much better computed solution! 2
The pivoting strategy adopted in this example is known as partial pivoting. Such a strategy exchanges rows so that all multipliers mik are less than or equal to 1 in magnitude.
4 Observe that in Example ?? m21 = 10 is very large compared to the sizes of the original elements of the linear system. As a result, elements of A1 (or Ak, in general) can grow unacceptably large. These, in turn, can result in loss of accuracy in fixed precision arithmetic, giving inaccurate results. Chap. 3. Matrix Computation CS414 Class Notes 48
Partial Pivoting Strategy. At the kth step of Gaussian elimination, find row i, i k, for which ≥ (k 1) (k 1) a − a − , j = k, . . . , n | ik | ≥ | jk | and exchange it with row k. This ensures that m < 1 at this step. Such a row permutation is applied | jk| at each step to ensure that mik < 1 for all k. In most cases this keeps the growth of elements of Ak, k = 1, 2, . . . , n 1 within acceptable| | limits. − Triangular Factorization with Partial Pivoting. To demonstrate this strategy, let us consider the LU decomposition of the matrix 1 0 1 A = 2 5 2 . 3 6 −9 We will also use a vector, IP IV , to keep trackof the row exchanges made during the factorization. The steps in the forward elimination proceed as follows. Step 1: Row 3 is the pivotal row since
(0) (0) max ai1 = a31 = 3; 1 i 3 ≤ ≤ | | therefore, we exchange rows 1 and 3, and initialize the first element of IP IV with 3. Now, we have
3 6 9 3 A˜0 = 2 5 2 , IP IV = . 1 0 −1 • • The transformed matrix A1 = M1A˜0 is given by
1 3 6 9 3 6 9 ˜ 2 A1 = M1A0 = 3 1 2 5 2 = 0 1 8 − 1 1 1 0 −1 0 2 −2 − 3 − − Storing the multipliers m and m in A , we get − 21 − 31 1 3 6 9 2 1 8 A1 = 3 − 1 3 2 2 − − Step 2: Again, row 3 is the pivotal row since
(1) (1) max ai2 = a31 = 2; 2 i 3 ≤ ≤ | | therefore, we exchange rows 2 and 3, and initialize the second element of IP IV with 3. Now, we have
3 6 9 3 2 2 2 A˜1 = 3 , IP IV = 3 . − − 1 3 1 8 • − The transformed matrix A2 = M2A˜1 is given by
1 3 6 9 3 6 9 A2 = M2A˜1 = 1 2 2 = 2 2 1 1 −1 −8 −0 −9 2 − − Chap. 3. Matrix Computation CS414 Class Notes 49
Storing the multiplier m in A , we get − 32 2 3 6 9 ˜ 2 2 2 A2 = 3 − − 1 1 3 2 9 − − Instead of the factorization A = LU, we have computed a factorization of a matrix whose rows have been permuted according to the vector IP IV . In other words, we have
P A = L U, ∗ where 1 3 6 9 0 0 1 2 L = 3 1 , U = 2 2 , P = 1 0 0 . 1 1 1 − −9 0 1 0 3 − 2 − Note that the permutation matrix Pis obtained by applying the row exchangesstored in IPIV to the identity matrix I. The exchanges must be applied sequentially in the order specified by IP IV (i), for i = 1, . . . , n 1. − 3 row to be exchanged with row 1 ← IP IV = 3 row to be exchanged with row 2 ← not used • ← Now, let us solve the linear system Ax = f, where A is given above, and
2 f = 5 . 18 We know that P Ax = P f, or LUx = P f, and so we solve
18 LUx = P f = ˜f = 2 . 5 Forward Sweep: Solve Ly = ˆf
1 y1 18 y1 = 18 1 3 1 y2 = 2 y2 + 6 = 2 y2 = 4 2 1 1 y 5 ⇒ y + 2 + 12 = 5 ⇒ y = −9 3 − 2 3 3 ⇒ 3 − Backward Sweep: Solve Ux = y
3 6 9 x1 18 x3 = 1 2 2 x2 = 4 2x2 2 = 4 x2 = 1 − −9 x −9 ⇒ 3−x +−6 + 9 =− 18 ⇒ x = 1 − 3 − 1 ⇒ 1 Therefore, 1 x = 1 . 1 Next, we give the MATLAB programs for the triangular factorization and solution of triangular systems. Chap. 3. Matrix Computation CS414 Class Notes 50
Triangular factorization with partial pivoting. function [L,U,IPIV] = GErowpiv(A); [n,n] = size(A); IPIV = 1:n; L = eye(n,n); for k=1:n-1 [maxv,r] = max(abs(A(k:n,k))); % max value in column q = r+k-1; % pivot row IPIV([k q]) = IPIV([q k]); % Exchange IPIV values A([k q],:) = A([q k],:); % Exchange rows of A if A(k,k) ~= 0 L(k+1:n,k) = A(k+1:n,k)/A(k,k); A(k+1:n,k+1:n) = A(k+1:n,k+1:n) - L(k+1:n,k)*A(k,k+1:n); end end U = triu(A);
Solving lower triangular system. function y = LowerTriSolve(L,f) n = length(f); y = zeros(n,1); for j=1:n-1 y(j) = f(j)/L(j,j); f(j+1:n) = f(j+1:n)-L(j+1:n,j)*y(j); end y(n) = f(n)/L(n,n);
Solving upper triangular system. function x = UpperTriSolve(U,y) n = length(y); x = zeros(n,1); for j=n:-1:2, x(j) = y(j)/U(j,j); y(1:j-1) = y(1:j-1)-x(j)*U(1:j-1,j); end; x(1) = y(1)/U(1,1); The MATLAB code that solves a linear system Ax = f using triangular factorization with partial pivoting is given as MATLAB [L,U,IPIV] = GErowpiv(A); % IPIV has row permutation y = LowerTriSolve(L,f(IPIV)); % f(IPIV) is permuted rhs x = UpperTriSolve(U,y);
Let us pay a closer look at GErowpiv(A). At stage k, after partial pivoting, let A˜k be of the form
! ! ! ! ! ! 1 o ! ! ! ! ! 1 (k) o o a ! ! ! ˜ 33 1 Ak = (k) , Mk = , o o a43 ! ! ! ! 1 (k) o o a ! ! ! ! 1 53 o o a(k) ! ! ! ! 1 63 Chap. 3. Matrix Computation CS414 Class Notes 51 i.e., U F I 0 A˜ = 1 1 M = 2 k 0 B˜ k 0 M˜ ( k ) ( k ) Therefore, U F A = M A˜ = 1 1 k+1 k k 0 M˜ B ( k k ) Let us focus on M˜ B : k ∗ k (k) 1 0 a(k) bT a bT 33 = 33 , (k) T m I3 c Ek a m + c E + mb ( ) ( ) 5 33 k 6
(k) Note that m = c/a33 ; in fact, m is computed by the MATLAB statement A(k+1:n,k)=A(k+1:n,k)/A(k,k). T− − Also, Ek + mb is computed by the statement A(k+1:n,k+1:n)=A(k+1:n,k+1:n)-A(k+1:n,k)*A(k,k+1:n).
T 1 Example 3.7 How should one compute α = c A− d ? Solution Since P A = LU, taking the inverse of both sides, we get
1 1 1 1 1 1 1 A− P − = U − L− A− = U − L− P. ⇒ T 1 1 Therefore, α = c U − L− P d. Let g = P d, solve Ly = g, • solve Ux = y, and • compute α = cT x. • Alternately, using MATLAB functions developed earlier, MATLAB [L,U,IPIV] = GErowpiv(A); g = d(IPIV); y = LowerTriSolve(L,g); x = UpperTriSolve(U,y); alpha = c’*x; 2
Iterative Refinement. An important technique to improve the accuracy of a computed solution relies on repeated solution of linear systems with successive residuals as the right hand side. In other words, one solves the linear system Ax(2) = f Ax(1), where x(1) is the initial solution. The procedure is outlined below: − Iterative Refinement 1. Compute P A = LU. 2. Solve LUx = P f = ˆf, i.e., Ly = ˆf, Ux = y xˆ [computed solution]. ⇒ 3. Let x(1) = xˆ; the residual is r = f Ax(1) = A(x x(1)) = Aδx(1). − − − 4. Solve Aδx(1) = r for error in solution. − 5. Compute improved solution: x(2) = x(1) δx(1) (repeat if needed) − Chap. 3. Matrix Computation CS414 Class Notes 52
Example 3.8 Consider the system Ax = f, i.e., 20 200000 x 200000 1 = 2 2 x 4 ( ) ( 2 ) ( ) The triangular factorization yields 1 0 20 200000 1 L = , U = , IP IV = 0.1 1 0 20000 ( ) ( − ) ( • ) Thus, the computed solution is 0 x(1) = . 1.0 ( ) Using 4-digit decimal arithmetic the residual is computed as 0 r(1) = f Ax(1) = . − 2.0 ( ) Solving Aδx(1) = r(1) for the error δx(1), we get − 1.0 1.0 δx(1) = − x(2) = x(1) δx(1) = . 0.0001 ⇒ − 0.9999 ( ) ( ) Compute r(2) and show it is much smaller than r(1). 2
3.4 Least Squares Problems
Least Squares Fitting. LS fitting problems arise in many applications. Consider a linear model for computing the height of a plant as a linear function of four nutrients, i.e.,
h = a1x1 + a2x2 + a3x3 + a4x4, where h is the plant height, and a1, a2, a3, and a4 are nutrient concentrations in the soil. The unknown parameters x1, x2, x3, and x4 can be determined from m observations of height and concentrations for m > 4. Typically, a large number of experiments are conducted to obtain the values for h, a1, a2, a3, and a4. Suppose the observations for the ith experiment are given by hi, ai1, ai2, ai3, ai4. If the model is perfect, then hi = ai1x1 + ai2x2 + ai3x3 + ai4x4 for i = 1, 2, . . . , m. In matrix notation we would then have h = Ax, or, h a a a a 1 11 12 13 14 x1 h a a a a 2 21 22 23 24 x2 . = . . . . , . . . . . x3 h a a a a x4 m m1 m2 m3 m4 where A is an m 4 matrix. In general, ho×wever, the model will not be perfect, making it impossible to find that vector x which exactly satisfies the above equation. In such situations, the main objective is to obtain an x for which
h Ax ( − (2 2 T is minimized. Recall that for a vector y, y 2 = y y. LS fitting arises also in the approximation( ( of known functions. Suppose we wish to approximate f(x) = √x on [0.25,1] via a linear approximation of the form
l(y) = α + βy Chap. 3. Matrix Computation CS414 Class Notes 53
Thus, given the table x x x x 1 2 · · · m √x f f f 1 2 · · · m in which fi = √xi, determine the best least squares fit parameters α and β. Let,
f1 1 x1 f2 1 x2 α r = f Aa = . . . , − . − . . β ( ) f 1 x m m then we seek to determine a so that r 2 = f Aa 2 is a minimum. ( (2 ( − (2 Least Squares Problem. The least squares problem is stated as follows:
min f Ax 2 x ( − (2 where A is an m n matrix with m n, and f, x are vectors of order m and n, respectively. Also, we assume that A has×linearly independen≥t columns, i.e., A is of full column-rank. We seek a factorization of A of the form,
R QT A = 0 ( ) T T where R is an n n upper triangular matrix, and Q is orthogonal, i.e., QQ = Q Q = Im. Since the vector 2-norm is invarian×t under orthogonal transformations, i.e.,
QT r 2 = (QT r)T (QT r) = rT QQT r = rT r = r 2 ( (2 ( (2 we have,
2 T 2 f Ax 2 = Q (f Ax) 2 ( − ( ( − ( 2 g R = 1 x g2 − 0 7( ) ( ) 72 7 2 7 7 g Rx 7 = 7 1 7 −g 7( 2 )72 7 7 7 7 This quantity is minimized for x = xˆ, for which R7xˆ = g1; then 7xˆ is called the least squares solution, and ˆr 2 = f Ax 2 = g2 2. ( (Next,( w−e describ( e(how( to construct the orthogonal factorization
R QT A = . 0 ( ) Consider the 2 2 orthogonal matrix × c s G = s c ( − ) in which c = cos θ and s = sin θ. It can be easily verified that
c2 + s2 0 1 0 GGT = GT G = = . 0 c2 + s2 0 1 ( ) ( ) In fact, given a vector x of size 2, we can construct G such that the vector y = Gx has zero as the second element. To do this, let c s x y 1 = 1 , s c x 0 ( − ) ( 2 ) ( ) Chap. 3. Matrix Computation CS414 Class Notes 54 then s x2 sx1 + cx2 = 0 = − ⇒ c x1 Using the identity c2 + s2 = 1, we have 2 x2 1 1 + 2 = 2 x1 c that further gives the following expression for c and s: x x c = 1 s = 2 2 2 2 2 x1 + x2 x1 + x2 Note that 8 8 y = cx + sx = x2 + x2 = x . 1 1 2 1 2 ( (2 Now, we can construct a 2 2 orthogonal matrix that9 introduces a zero in the second position of a vector of size 2. We use this idea to in×troduce zeros in the lower triangular part of A to obtain the required upper triangular form by repeated multiplication with suitable 2 2 orthogonal matrices. The matrix × 1 . .. (1) G = 1 m 1,m − (1) (1) cm 1,m sm 1,m (1)− (1)− sm 1,m cm 1,m − − − is used to introduce a zero in the last position, i.e., mth location in the first column. After that, the matrix
1 . .. (1) 1 Gm 2,m 1 = (1) (1) − − cm 2,m 1 sm 2,m 1 (1)− − (1)− − sm 2,m 1 cm 2,m 1 − − − − − 1 is used to introduce a zero in the (m 1)th position of the first column. The process of introducing zeros in − (1) the first column is continued by forming the matrix Gj 1,j , for j = m 2, m 3, . . . , 2, that introduce zeros T − − − in the jth location. Let us define Q1 to be the product of these matrices, i.e.,
T (1) (1) (1) Q1 = G1,2 Gm 2,m 1Gm 1,m · · · − − − The same technique can be used to zero out the elements below the diagonal in the second column, followed by the third column all the way up to the last column. Note that introducing zeros in this manner (k) doesn’t effect the zeros introduced earlier. Let Gj 1,j be the matrix used to zero out the element in the (j, k) position. Let us also define −
T (2) (2) (2) Q2 = G2,3 Gm 2,m 1Gm 1,m · · · − − − . . . . T (n) (n) (n) Qn = Gn 1,n Gm 2,m 1Gm 1,m, − · · · − − − then, R QT QT QT A = . n · · · 2 1 0 ( ) Chap. 3. Matrix Computation CS414 Class Notes 55
(k) T Note that since each Gj 1,j is orthogonal, each Qk is also orthogonal. Finally, defining − QT = QT QT QT , n · · · 2 1 we have the required orthogonal matrix that transforms A into an upper triangular matrix. The same transformations applied to f give g = QT QT QT f. n · · · 2 1 T For example, if A is a 5 3 matrix (m=5, n=3), Q1 introduces zeros below the diagonal in the first T × column of A, Q2 introduces zeros below the diagonal in the second column, and so on until
R QT QT A = . n · · · 1 0 ( ) T In other words, zeros in locations 1, 2, 3, and 4 are introduced by Q1 , zeros in locations 5, 6, and 7 are T T introduced by Q2 , and zeros in locations 8 and 9 are introduced by Q3 .
! ! ! 4 ! ! 3 7 ! 2 6 9 1 5 8 MATLAB function [xLS,res] = LSq (A,f) [m,n] = size(A); for j=1:n, for i=m:-1:j+1 x = [A(i-1,j),A(i,j)]; [c,s] = x/norm(x); A(i-1:i,j:n) = [c s; -s c]*A(i-1:i,j:n); f(i-1:i) = [c s; -s c]*f(i-1:i); end; end; xLS = UpperTriSolve(A(1:n,1:n), f(1:n)) if m==n, res = 0; else res = norm(f(n+1:m)); end;