Chapter 3 Matrix Computation The main objective of this chapter is to solve the linear system Ax = f, where A is a square n n matrix, and x and f are vectors of order n. × 3.1 Basics Matrix. a a a 11 12 · · · 1n a21 a22 a2n A = . · · · . = [aij ] i, j = 1, 2, ..., n. a a a n1 n2 · · · nn Column Vectors. x1 f1 x2 f2 x = . , f = . . x f n n Row Vectors. T x = (x1, x2, . , xn) T f = (f1, f2, . , fn) Inner Products. b1 n b2 T a b = (a1, a2, . , an) . = aibi scalar . ← i=1 b ' n Operation count: n multiplications, n 1 additions 2n operations. − $ 29 Chap. 3. Matrix Computation CS414 Class Notes 30 Outer Products. a a b a b a b 1 1 1 1 2 · · · 1 n a2 a2b1 a2b2 a2bn T ab = . (b1, b2, . , bn) = . · · · . matrix . ← a a b a b a b n n 1 n 2 · · · n n Operation count: n2 multiplications = n2 operations. 3.1.1 Setting up matrix problems in MATLAB Hilbert matrix. Suppose we want to construct a matrix A s.t. 1 A = ij i + j 1 − MATLAB A = zeros(n,n); for i = 1:n, for j = 1:n, A(i,j) = 1/(i+j-1); end; end; This is known as a Hilbert matrix, and can also be created in MATLAB by A = hilb(n). For n = 5, this matrix is of the form 1 1/2 1/3 1/4 1/5 1/2 1/3 1/4 1/5 1/6 A = 1/3 1/4 1/5 1/6 1/7 1/4 1/5 1/6 1/7 1/8 1/5 1/6 1/7 1/8 1/9 Observing that the Hilbert matrix is symmetric, the above MATLABcode fragment could be modified to take advantage of this symmetry: MATLAB A = zeros(n,n); for i = 1:n, for j = i:n, A(i,j) = 1/(i+j-1); A(j,i) = A(i,j); end; end; Another matrix. As another example of matrix set-up, consider the following MATLAB code: MATLAB P = zeros(n,n); P(:,1) = ones(n,1); % P(:,1) = 1st column of P for i = 2:n, for j = 2:i, P(i,j) = P(i-1,j-1)+P(i-1,j); end; end; Chap. 3. Matrix Computation CS414 Class Notes 31 For n = 5, the above code generates the following matrix 1 0 0 0 0 1 1 0 0 0 P = 1 2 1 0 0 1 3 3 1 0 1 4 6 4 1 Next, we present two special matrices that can be created given a vector of n components, i.e., the elements of the matrix depend only on n parameters. Vandermonde matrix. This a special type of matrix that can be created from a vector of n components. Given a vector x1 x2 x = . . x n we can create a matrix V as follows: MATLAB n = length(x); V(:,1) = ones(n,1); for j = 2:n, V(:,j) = x .* V(:,j-1); end; Over here, the jth column of V is obtained by element-wise multiplication of the vector x with the (j 1)th column. For n = 4, V is of the form − 2 3 1 x1 x1 x1 2 3 1 x2 x2 x2 V = 2 3 1 x3 x3 x3 2 3 1 x4 x x 4 4 Circulant matrix. Another matrix that can be created from a vector of size n is the circulant matrix. Given a vector a1 a2 a = . . a n we can create a circulant matrix C as follows: MATLAB function C = circulant(a) n = length(a,1); C(1,:) = a’; for j = 2:n, C(i,:)=[C(i-1,n) C(i-1,1:n-1)]; end; Over here, the ith row of C is obtained by permuting (i 1)th row; the last element of (i 1)th row is permuted to the start of the ith row. For n = 4, C is of the− form − a1 a2 a3 a4 a a a a C = 4 1 2 3 a3 a4 a1 a2 a2 a3 a4 a1 Chap. 3. Matrix Computation CS414 Class Notes 32 3.1.2 Structure of Matrices The nonzero elements of a matrix determine its structure. Following are commonly occurring matrices. ! 0 0 0 0 ! ! 0 0 0 0 ! 0 0 0 ! ! ! 0 0 0 0 ! 0 0 0 ! ! ! 0 0 0 0 ! 0 0 0 ! ! ! 0 0 0 0 ! 0 0 0 ! ! Diagonal Tridiagonal ! 0 0 0 0 ! ! ! ! ! ! ! ! ! ! ! ! 0 0 0 0 ! ! ! ! ! ! ! ! ! ! ! ! 0 0 0 0 ! ! ! 0 ! ! ! ! ! ! ! ! 0 0 0 0 ! ! 0 0 ! ! ! ! ! ! ! ! 0 0 0 0 ! 0 0 0 ! ! Lower triangular Upper triangular Upper Hessenberg A diagonal matrix can be constructed from a vector d, where d = [d d d d ]; 1 2 3 · · · n by the MATLAB statement D = diag(d). This constructs the diagonal matrix D, s.t., d1 d2 D = . .. d n MATLAB also provides a way to specify diagonal entries other than the main diagonal. For example, consider the following code: MATLAB m = 3; k = 2; v = [10 20 30]; A = diag(v,k); This constructs a matrix A with the vector v forming the kth diagonal of A: 0 0 10 0 0 0 0 0 20 0 A = 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 Also, the statement v = diag(A,k) extracts the kth diagonal in the form of a vector. A negative k refers to diagonals below the main diagonal while k = 0 refers to the main diagonal. Block structures. Matrices with block structure can also be constructed in MATLAB. A block structured matrix consists of elements that are in turn smaller matrices. For example, a block structured matrix of the form A11 A12 (3 3) (3 2) × × A = A21 A22 (2 3) (2 2) × × can be constructed from the following code MATLAB Chap. 3. Matrix Computation CS414 Class Notes 33 A11 = rand(3,3); A12 = rand(3,2); A21 = [1 2 3 ; 4 5 6]; A22 = [7 8 ; 9 10]; A = [A11 A12 ; A21 A22]; 3.2 Matrix Operations Matrix-Vector Multiplication. Given an m n matrix A and a vector x of size n, we wish to compute the vector y of size m s.t. × y = Ax where a a a x 11 12 · · · 1n 1 a21 a22 a2n x2 A = . · · · x = . . a a a x m1 m2 · · · mn n gives the vector y with elements n yi = aikxk i = 1, 2, . , m 'k=1 Operation count: mn multiplications, m(n 1) additions 2mn operations. − $ MATLAB y = zeros(m,1); for i = 1:m, for j = 1:n, y(i) = y(i) + A(i,j) * x(j) end; end; This can be computed in a number of ways. Alternative 1: (row-oriented) T T a1 x1 a1 x T T a2 x2 a2 x A = . , x = . y = . . ⇒ . aT x aT x m n m The corresponding MATLAB program uses m inner products: MATLAB for i = 1:m, y(i) = A(i,:) * x; end; Alternative 2: (column-oriented) x1 n x2 A = (b1, b2, . , bn), x = . y = xjbj . ⇒ j=1 x ' n The corresponding MATLAB program uses summation of n vectors, i.e., “scalar * vector + vector”: MATLAB Chap. 3. Matrix Computation CS414 Class Notes 34 for j = 1:n, y = y + x(j) * A(:,j); end; Matrix-Matrix Multiplication. Given an m r matrix A and an r n matrix B we wish to compute the product C of size m n s.t. × × × C = A B ∗ (m ↓ n) (m ↓ r) (r ↓ n) × × × The matrix C can be represented as i = 1, 2, . , m C = [c ], ij j = 1, 2, . , n where r cij = aikbkj k'=1 Operation count: mnr multiplications, mn(r 1) additions 2mnr operations. − $ MATLAB C = zeros(m,n); for j = 1:n, for i = 1:m, for k = 1:r, C(i,j) = C(i,j) + A(i,k)*B(k,j); end; end; end; Alternative 1: (inner products) T a1 aT 2 T A = . , B = (b1, b2, . , bn) cij = ai bj . ⇒ aT m MATLAB for j = 1:n, for i = 1:m, C(i,j) = A(i,:)*B(:,j); end; end; Alternative 2: (outer products) T h1 hT r 2 T A = (g1, g2, . , gr) B = . , C = gkhk m n matrix . ⇒ ← × k=1 hT ' r MATLAB Chap. 3. Matrix Computation CS414 Class Notes 35 for k = 1:r, C = C + A(:,k)*B(k,:); end; Alternative 3: (matrix-vector products) B = (b , b , . , b ) C = (Ab , Ab , . , Ab ) 1 2 n ⇒ 1 2 n MATLAB for j = 1:n, C(:,j) = A*B(:,j); end; 3.2.1 Matrix Norm, Inverse, and Sensitivity of Linear Systems Inverse of a Matrix. If A is a square matrix of order n, then A is nonsingular if there exists a unique n n matrix X such that × 1 1 AX = XA = I = . .. 1 1 The inverse, X, is also denoted by A− . The following property holds for inverse of a product of two matrices: 1 1 1 (A B)− = B− A− . ∗ ∗ Note also that (A B)T = BT AT . ∗ ∗ The identity matrix I is often expressed as T I = (e1, e2, . , en), ei = (0, 0, . , 0, 1, 0, . , 0), i.e., ei has a one in the ith position and zero everywhere else. Vector and matrix norms. Suppose that x" and x"" are two approximations of x. Let 1 1.01 1 x = 1 , x" = 1.01 , x"" = 1.02 .