CS 515: Homework 1
Yingwei Wang ∗ Department of Mathematics, Purdue University, West Lafayette, IN, USA
1 Matrix product
1.1 (a)
1 3 2 1 2 ∗ 3, 2, 1 = 6 4 2 . 3 9 4 3 1.2 (b)
1 3, 2, 1 ∗ 2 = 10. 3 1.3 (c)
0 0 1 1 2 3 7 8 9 0 1 0 ∗ 4 5 6 = 4 5 6 . 1 0 0 7 8 9 1 2 3 2 Symmetric matrices
Give an example to show that the product of two symmetric matrices need not to be symmetric.
∗E-mail address: [email protected]; Tel: 765 237 7149
1 Yingwei Wang Numerical Linear Algebra
1 0 0 1 0 1 1 0 1 0 2 0 ∗ 0 1 0 = 0 2 0 . 0 0 3 1 0 1 3 0 3 3 Elementary reflector
Let H = I − 2xxT , where xT x = 1.
3.1 H is symmetric
HT = IT − 2(xxT )T = I − 2xxT = H. So H is symmetric.
3.2 H is involuntary
H2 =(I − 2xxT )(I − 2xxT ) = I2 − 4xxT +4xxT xxT = I − 4xxT +4x(xT x)xT = I − 4xxT +4xxT = I.
So H is involuntary.
3.3 H is orthogonal Since H is symmetric and involuntary, we have
HT H = H2 = I, HHT = H2 = I.
So H is orthogonal.
Remark 3.1. H is often called “elementary reflector ”or a “Householder transformation”.
2 Yingwei Wang Numerical Linear Algebra
4 Property about zero vectors
Proposition 4.1. Let A ∈ Rm×n. Show that xT AT Ax =0 if and only if Ax =0. Proof. If Ax = 0, then it is obvious that xT AT Ax = 0. T Conversely, let y = Ax =(y1, ··· ,yj, ··· ,ym) be a column vector, then xT AT Ax =0, ⇒ yT y =0, m 2 ⇒ yj =0, j=1 X ⇒ yj =0, ∀j, ⇒ y =0, ⇒ Ax =0.
5 Block matrices
Assuming the partition conform, compute
5.1 (a)
(X,Y )T ∗ A ∗ (X,Y ) XA = ∗ (X,Y ) Y A XAX XAY = . Y AX Y AY 5.2 (b)
β bT α (α, xT ) ∗ ∗ b B x α = (αβ + xT b, αbT + xT B) ∗ x = α2β + αxT b + αbT x + xT Bx.
3 Yingwei Wang Numerical Linear Algebra
5.3 (c)
α aT β bT ∗ a A b B αβ + aT b αbT + aT B = . βa + Ab abT + AB 6 Matrix equation
n T T T Proposition 6.1. Let x ∈ R be partitioned in the form x = (ξ1,y ) where y = n−1 (ξ2,ξ3, ··· ,ξn). Show that if ξi =06 , there is a unique vector b ∈ R such that
1 0 ξ ξ ∗ 1 = 1 (6.1) b I − y 0 n 1
Proof. Choose b = −y/ξi, then
1 0 ξ ∗ 1 b In− y 1 ξ = 1 ξb + y ξ = 1 0
If there are two vectors b1, b2 satisfying (6.1), then
ξ1b1 + y =0, ξ1b2 + y =0,
⇒ ξ1(b1 − b2)=0,
⇒ b1 = b2.
7 Nonsingular matrix
Proposition 7.1. Let A = diag (λ1,λ2, ··· ,λn). Show that A is nonsingular if and only if λi =06 for 1 ≤ i ≤ n.
4 Yingwei Wang Numerical Linear Algebra
n Proof. Since det(A) = Πi=1λi, we know that if A is nonsingular then det(A) =6 0, which implies that λi =06 for 1 ≤ i ≤ n. Conversely, if λi =06 for 1 ≤ i ≤ n, then det(A) =6 0, which means A is nonsingular. −1 −1 −1 −1 Besides, then inverse of A is A = diag (λ1 ,λ2 , ··· ,λn ).
8 Inverse
Proposition 8.1. Let u, v ∈ Rn and σ =6 0. Suppose that vT u − σ−1 =6 0. Show that (I − σuvT ) is nonsingular and has an inverse given by (I − τuvT ) where σ−1 + τ −1 = vT u.
T T Proof. Let An = In − σuv and Dn = det(An). Let u = (u1,u2, ··· ,un) and v = T (v1, v2, ··· , vn) . Then by the knowledge of linear algebra, it is easy to know that
1 − σu1v1 −σu1v2 ··· −σu1vn −σu v 1 − σu v ··· −σu v det(A ) = det 2 1 2 2 2 n (8.1) n ··· −σunv −σunv ··· 1 − σunvn 1 2 1 − σu1v1 −σu1v2 ··· 0 1 − σu1v1 −σu1v2 ··· σu1vn −σu v 1 − σu v ··· 0 −σu v 1 − σu v ··· σu v = det 2 1 2 2 − det 2 1 2 2 2 n(8.2) ··· ··· −σunv −σunv ··· 1 −σunv −σunv ··· σunvn 1 2 1 2 1 0 ··· 0 0 1 ··· 0 = D − − det (8.3) n 1 ··· −σunv −σunv ··· σunvn 1 2 = Dn−1 − σunvn. which means Dn = Dn−1 − σvnun. (8.4) Besides, we know that D1 =1 − σu1v1. Now we can get
n T Dn =1 − σ ujvj =1 − σv u. j=1 X
5 Yingwei Wang Numerical Linear Algebra
Since vT u − σ−1 =6 0, we know that det(I − σuvT ) =6 0, which implies that (I − σuvT ) is nonsingular. Now we want to find the inverse of (I − σuvT ). Choose τ such that 1 1 + = vT u, σ τ σ + τ ⇒ = vT u, στ ⇒ σ + τ = στvT u. Consider the product (I − τuvT )(I − σuvT ): (I − τuvT )(I − σuvT ) = I − (σ + τ)uvT + στ(uvT )2, = I − στvT uuvT + στ(uvT )2. T T T 2 Let X = v uuv =(xpq)n×n, Y =(uv ) =(ypq)n×n, then n
xpq = ukvk upvq, k=1 ! nX
ypq = (upvk)(ukvq) k=1 Xn
= ukvk upvq. k ! X=1 It implies that X = Y and further (I − τuvT )(I − σuvT )= I. Similarly, we can prove that (I − σuvT )(I − τuvT )= I. Then we are done.
Remark 8.1. From (8.2) to (8.3), we assume that un and vn are not 0. But if un = 0 or v0 =0, then (8.4) is still true. So, anyway, (8.4) is true. Besides, from (8.2) to (8.3), if vj =0, then it is fine; if vj =06 , then we can use the last T row to make jth row become ej = (0, ··· , 0, 1, 0, ··· , 0) where 1 is in the jth place. Finally, from (8.1) to (8.2), it is just a common trick for computing the det(An). You can find that in any textbook on linear algebra.
9 Matrices
Proposition 9.1. Let B ∈ Rn×n,S ∈ Rk×k and U, V ∈ Rn×k. Assuming that B,S and [V T B−1U − S−1] are nonsingular, show that (B − USV T )−1 = B−1 − B−1UTV T B−1,
6 Yingwei Wang Numerical Linear Algebra where S−1 + T −1 = V T B−1U.
Proof. On one hand,
(B − USV T )(B−1 − B−1UTV T B−1), = I − UTV T B−1 − USV T B−1 + USV T B−1UTV T B−1, = I − U(T + S)V T B−1 + US(S−1 + T −1)TV T B−1, = I.
On the other hand,
(B−1 − B−1UTV T B−1)(B − USV T ), = I − B−1USV T − B−1UTV T + B−1UTV T B−1USV T , − − − − = I − B 1U(S + T )V T + B 1UT (S 1 + T 1)SV T , = I.
Then we are done.
Question: A ∈ Rn×n is hermitian if AH = A. If A = B + iC, then it is easy to show that BT = B and CT = −C. Suppose that we represent A in an array A.herm with the property that A.herm(i, j) houses bij if i ≥ j and cij if j > i. Using this data structure write a matrix-vector multiply function that computes ℜ(z) and ℑ(z) from ℜ(x) and ℑ(x) so that z = Ax. Solution: Let x =(xj) be a column vector, and each element xj = pj + iqj, where pj = ℜ(xj) and qj = ℑ(xj). Let A = B + iC be hermitian and A.herm =(ajk), B =(bjk), C =(cjk). We know that
ajk = bjk if j ≥ k, & ajk = cjk if j ≤ k.
Besides, bjk = bkj, & cjk = −ckj.
7 Yingwei Wang Numerical Linear Algebra
Let z =(zj) be a column vector. Then from z = Ax, we know that
n
zj = (bjk + icjk)(pk + iqk) k=1 Xn
= (bjkpk − cjkqk)+ i(cjkpk + bjkqk). k=1 X j n j n
⇒ ℜ(zj)= ajkpk − akjpk − akjqk + ajkqk, (10.1) k k j k k j X=1 X= +1 X=1 X= +1 j n j n
ℑ(zj)= − akjpk + ajkpk + ajkqk − akjqk. (10.2) k k j k k j X=1 X= +1 X=1 X= +1 From the formula (10.1) and (10.2), we can write some code using any language.
11 The algebra of triangular matrices
Proposition 11.1. Here are some properties about the products and inverses of triangular and unit triangle matrices.
1. The inverse of an upper (lower) triangular matrix is upper (lower) triangular.
2. The product of two upper (lower) triangular matrices is upper (lower) triangular.
3. The inverse of a unit upper (lower) triangular matrix is upper (lower) triangular.
4. The product of two unit upper (lower) triangular matrices is upper (lower) triangular.
Proof. (1). Let Tn×n be the upper triangular matrix, then T (i, j) = 0 if i > j. Induction by n. If n = 1, then it is obviously true. −1 Suppose T(n−1)×(n−1) is an upper triangular matrix, then solving the equation TS = I, we can get
S(n, n)=1/T (n, n), S(n, 1 : n − 1)=0, S(1 : n − 1, n)= −T (1 : n − 1, 1 : n − 1)−1T (1 : n − 1, n)S(n, n), S(1 : n − 1, 1 : n − 1) = T (1 : n − 1, 1 : n − 1)−1, which is triangular by assumption.
It indicates that Sn×n is also an upper triangular matrix.
8 Yingwei Wang Numerical Linear Algebra
(2). Let An×n and Bn×n be two upper triangular matrices, then A(i, j)=0, B(i, j)=0 if i > j. Let C = AB. Since A(i, k)B(k, j) = 0 whenever k < i or j
S(n, n)=1/T (n, n)=1, S(n, 1 : n − 1)=0, S(1 : n − 1, n)= −T (1 : n − 1, 1 : n − 1)−1T (1 : n − 1, n)S(n, n), S(1 : n − 1, 1 : n − 1) = T (1 : n − 1, 1 : n − 1)−1, which is unit triangular by assumption.
It indicates that Sn×n is also a unit upper triangular matrix. (4). Similarly as Part (2). Let An×n and Bn×n be two unit upper triangular matrices, then A(i, j)=0, B(i, j) = 0 if i > j. Besides, A(k,k) = B(k,k)=1, ∀1 ≤ k ≤ n. Let C = AB. Since A(i, k)B(k, j) = 0 whenever k < i or j
9 CS 515: Homework 2
Yingwei Wang ∗ Department of Mathematics, Purdue University, West Lafayette, IN, USA
1 Matrix inverse
Question: Show that if R = D − U, where D is a nonsingular diagonal matrix of order n and U is strictly upper triangular, then R−1 = D−1[I + UD−1 +(UD−1)2 + ··· +(UD−1)n−1]. (1.1) Solution: Just do the matrix operations, we can get D−1[I + UD−1 +(UD−1)2 + ··· +(UD−1)n−1](D − U), = [D−1 + D−1UD−1 + D−1(UD−1)2 + ··· + D−1(UD−1)n−1](D − U), = [I + D−1U +(D−1U)2 + ··· +(D−1U)n−1] − [D−1U +(D−1U)2 + ··· +(D−1U)n−1 +(D−1U)n] = I − (D−1U)n. Since D is a nonsingular diagonal matrix and U is strictly upper triangular, we know that D−1U is also strictly upper triangular. Recall that every strictly upper (or lower) triangular matrix is nilpotent. It follows that (D−1U)n =0. Now we know that D−1[I + UD−1 +(UD−1)2 + ··· +(UD−1)n−1](D − U)= I. Similarly, we can also prove (D − U)D−1[I + UD−1 +(UD−1)2 + ··· +(UD−1)n−1]= I. It implies that R−1 = D−1[I + UD−1 +(UD−1)2 + ··· +(UD−1)n−1], where R = D − U. ∗E-mail address: [email protected]; Tel: 765 237 7149
1 Yingwei Wang Numerical Linear Algebra
2 Inverse of lower triangular matrix
Question: Write a Matlab code to generate the inverse of a lower triangular matrix, one row at a time. Solution: function b = lower_inv(a) %%% a is a lower triangluar matrix %%% b is the inverse of a
%% prepare nn = size(a); b = zeros(nn); n = nn(1);
%% main loop for i = 1:n if a(i,i)==0, fprintf(’This is a singular matrix. No inverse.’) b=NaN; break; else if i==1 b(i,i) = 1/a(i,i); else pa = a(i,1:i-1); pb = b(1:i-1,1:i-1); b(i,1:i-1) = -pa*pb/a(i,i); b(i,i) = 1/a(i,i); end end end
Question: Let A be symmetric with a11 =6 0. Show that after one step of Gaussian elimina- tion (without partial pivoting), we obtain a T a11 a 0 Aˆ 2 Yingwei Wang Numerical Linear Algebra where Aˆ is also symmetric. Solution: Let A =(aij)1≤i,j≤n. That A is symmetric means
aij = aji, ∀ i, j. (3.1)
Suppose after one step of Gaussian elimination, A becomes A˜ = (˜aij)1≤i,j≤n. Then for 2 ≤ i, j ≤ n, we know that
a1jai1 a˜ij = aij − , a11 a1iaj1 a˜ji = aji − . a11 By Eq.(3.1), we know that a˜ij =˜aji, ∀ 2 ≤ i, j ≤ n. It implies that Aˆ is also symmetric. Remark 3.1. If we show this in blocked matrix form rather than element form, then it may be more clearly. Let a aT A = 11 1 a A(0) 1 22 Then after one step of Gaussian elimination, we obtain
T (1) a11 a1 A = T , 0 A(0) − a1a1 " 22 a11 # which means a aT ˆ (0) 1 1 A = A22 − . a11 It is obvious that Aˆ is also symmetric.
4 Diagonally dominant matrix
Question: Let A ∈ Rn×n be diagonally dominant, i.e.
|aii| > |aji|, i =1, 2, ··· , n. (4.1) j6 i X= Show that
3 Yingwei Wang Numerical Linear Algebra
1. any principal submatrix of A is diagonally dominant; 2. A is nonsingular. Solution:
1. Let P =(pkl)m×m be a principal submatrix of A. By definition, P is obtained from A by removing any n − m rows and the same n − m columns. So for each k,
pkk = aii, where i = i(k) is dependent on k. For each pair of i and k, it is easy to know that
|pkl| ≤ |aji|. k6 l j6 i X= X= By (4.1), we know that
|pkk| = |aii| > |aji| ≥ |pkl|, j6 i k6 l X= X= which means P is also diagonally dominant. 2. Suppose A is singular, we want to get a contradiction. T If A is singular, then Ax = 0 has a nonzero solution x =(x1, ··· , xn) . Let
|xk| = max |xj|, 1≤j≤n
then |xk| > 0. n By j=1 akjxj = 0, we can get that n P | akjxj| j6 k X= = | − akkxk|
= |akk||xk|
> |xk| |ajk| j6 k X= ≥ |ajk||xj| j=6 k Xn
≥ | akjxj|, j6 k X= which is a contradiction.
4 Yingwei Wang Numerical Linear Algebra
5 LDU decomposition
Question: Let A be symmetric and nonsingular, and let
A = LDU be the “LDU decomposition ”of A. Show that U = LT . Solution: Suppose A = LDU, where L is unit lower triangular, U is unit upper triangular and D is diagonal. Since A = AT , we can get LDU = U T DLT . Besides, A is nonsingular, so L and LT are also nonsingular. Then we can get
DU(LT )−1 = L−1U T D. (5.1)
On one hand, U(LT )−1 is upper triangular, so the left hand side of Eq.(5.1) is upper triangular. On the other hand, L−1U T is lower triangular, so the right hand side of Eq.(5.1) is lower triangular. Now we know that each side of Eq.(5.1) is diagonal. Furthermore, that U and (LT )−1 are unit upper triangular means U(LT )−1 is unit upper triangular; similarly, that L−1 and U T are unit lower triangular means L−1U T is unit lower triangular. Then we can get each side of Eq.(5.1) is equal to D. It follows that
U(LT )−1 = L−1U T = I, ⇒ L = U T .
6 Positive definite
Question: Show that if A is symmetric, diagonally dominant, and has positive diagonal elements, then A is positive definite. Solution: Let λ ≤ 0 be a non-positive real number, and B = A − λI where I is a identity matrix. Then B is also a symmetric and diagonally dominant matrix. By the conclusion in Section 4 (any diagonally dominant matrix is nonsingular), we can know that B is nonsingular, which means |B|= 6 0. It follows that every non-positive number must not be the eigenvalue of A, which means all of the eigenvalues of A are positive. It implies that A should be positive definite.
5 CS 515: Homework 3
Yingwei Wang ∗ Department of Mathematics, Purdue University, West Lafayette, IN, USA
1 LU-factorization
Question: Write a Matlab code for the LU-factorization, with partial pivoting, of a matrix A ∈ Rn×n. Use this code to show that for 2 1 −2 A = 2 4 −9 , (1.1) 44 2
1 0 0 4 4 2 P A = 0.5 1 0 0 2 8 0.5 −0.5 1 0 0 1 where P is that permutation matrix corresponding to partial pivoting.
Solution: function [L,U,P]=lu_wyw(A) %%% LU factorization with partial (row) pivoting %%% A: an n \times n square matrix, %%% L: unit lower triangular matrix %%% U: upper triangular %%% P: permutation matrix such that PA=LU
%% prepare [n,n]=size(A); L=eye(n); P=L; U=A;
∗E-mail address: [email protected]; Tel: 765 237 7149
1 Yingwei Wang Numerical Linear Algebra
%% main loop for k=1:n % choose the pivot [pivot m]=max(abs(U(k:n,k))); m=m+k-1; if m~=k % interchange rows m and k in U temp=U(k,:); U(k,:)=U(m,:); U(m,:)=temp; % update the permutation matrix temp=P(k,:); P(k,:)=P(m,:); P(m,:)=temp; if k >= 2 temp=L(k,1:k-1); L(k,1:k-1)=L(m,1:k-1); L(m,1:k-1)=temp; end end for j=k+1:n L(j,k)=U(j,k)/U(k,k); U(j,:)=U(j,:)-L(j,k)*U(k,:); end end
Let A be the same as (1.1), and use the function [L, U, P ] = lu wyw(A) in Matlab, we can get
1.0000 0 0 L = 0.5000 1.0000 0 , 0.5000 −0.5000 1.0000 4 4 2 U = 0 2 8 , 0 0 1 0 0 1 P = 0 1 0 . 1 0 0
2 Yingwei Wang Numerical Linear Algebra
2 Forward- and back-sweeps
Question: Write Matlab codes for forward- and back-sweeps to solve the linear system
Ax = f, (2.1) where A is given in the first section and f = [1, 1, 1]T .
Solution: The Matlab function for forward-sweeps is as follows: function y = forward_wyw(L,b) %%% Solving a lower triangular system by forward-sweep %%% L: an n \times n lower triangular matrix %%% b: an n \times 1 vector %%% y: the solution to the linear system L y = b.
%% prepare n=length(b); y=zeros(n,1);
%% main loop for j=1:n if L(j,j)==0 fprintf(’This matrix is singular.’); break; end; y(j)=b(j)/L(j,j); b(j+1:n)=b(j+1:n)-L(j+1:n,j)*y(j); end
The Matlab function for back-sweeps is as follows: function x = back_wyw(U,b) %%% Solving an upper triangular system by back-sweep %%% U: an n \times n upper triangular matrix %%% b: an n \times 1 vector %%% x: the solution to the linear system Ux = b.
%% prepare n=length(b); x=zeros(n,1);
%% main loop
3 Yingwei Wang Numerical Linear Algebra for j=n:-1:1 if (U(j,j)==0) fprintf(’This matrix is singular.’); break; end; x(j)=b(j)/U(j,j); b(1:j-1)=b(1:j-1)-U(1:j-1,j)*x(j); end
Now let us solve the linear system (2.1). First, use [L, U, P ] = luwyw(A). The results are shown in Section 1. Second, solve Ly = P f by y = forward wyw(L, p ∗ f). The result is
1.0000 y = 0.5000 . 0.7500
Third, solve Ux = y by x = back wyw(U, y). The result is
2.6250 x = −2.7500 . 0.7500
3 Cholesky factorization
Question: Write a Matlab code for the Cholesky factorization of a symmetric positive definite matrix B ∈ Rn×n, and use it together with appropriate codes from Section 2 to solve the system By = g, (3.1) where 4 2 1 1 B = 2 6 3 , g = −7 . 1 3 8 3
Solution: The Matlab function for Cholesky factorization of a symmetric positive definite matrix is as follows: function L=cholesky_wyw(A) %%% Cholesky Factorization for symmetric positive definite matrix %%% Find a lower triangular matrix L such that A = L*L’,
%% prepare
4 Yingwei Wang Numerical Linear Algebra
[n nn]=size(A);
%% main loop for k=1:n A(k,k)=sqrt(A(k,k)); A(k+1:n,k)=A(k+1:n,k)/A(k,k); for j=k+1:n A(j:n,j)=A(j:n,j)-A(j,k)*A(j:n,k); end end L=tril(A);
Now let us solve the linear system (3.1). First, do the Cholesky factorization by L = cholesky wyw(B). We can get
2.0000 0 0 L = 1.0000 2.2361 0 . 0.5000 1.1180 2.5495
Second, solve Ly = g by y = forward wyw(L, g). The result is
0.5000 y = −3.3541 . 2.5495
Third, solve LT x = y by x = back wyw(L′,y). The result is
1.0000 x = −2.0000 . 1.0000
5 CS 515: Homework 4
Yingwei Wang ∗ Department of Mathematics, Purdue University, West Lafayette, IN, USA
1 Elementary reflector
Question: Let U = I − 2uuT be an elementary reflector. Let x be given and let x = v + w where v lies along u and w is orthogonal to u. Show that Ux = −v + w. Interpret this result geometrically in Rn. Solution: We know that
U = I − 2uuT is an elementary reflector, 2 ⇒ 2= π−1 = , uT u ⇒ uT u =1.
Besides,
v k u ⇒ v = cu, c is a scalar constant, w ⊥ u ⇒ uT w =0.
Now, since x = v + w and U = I − 2uuT , we know that
Ux =(I − 2uuT )x, = x − 2uuT x, =(v + w) − 2uuT v − 2uuT w, =(v + w) − 2uuT (cu) − 2u0, =(v + w) − 2cu(uT u), =(v + w) − 2v, = −v + w. ∗E-mail address: [email protected]; Tel: 765 237 7149
1 Yingwei Wang Numerical Linear Algebra
2 New elementary reflector
n Question: Show that if x ∈ R , there is an elementary reflector U such that Ux = −σen. Derive algorithms for computing U. T T Solution: Given x = (ξ1,ξ2, ··· ,ξn) , then let σ = ±kxk2, u = x + σen, π = u u/2. Define U = I − π−1uuT .
Claim that Ux = −σen. First, we can rewrite π as 1 π = uuT , 2 1 = (x + σe )T (x + σe ), 2 n n 1 = (kxk2 + σ2 +2σξ ), 2 2 n 2 = σ + σξn.
Second, we can compute Ux = −σen.
Ux =(I − π−1uuT )x, = x − π−1uuT x, −1 T T = x − π (x + σen)(x x + σen x), −1 = x − π (kxk2 + σξn)(x + σen),
= x − (x + σen),
= −σen.
The algorithm for computing U is as follows:
T 1. Given ~x =(ξ1,ξ2, ··· ,ξn) ;
2. Compute σ = (sign ξn) kxk2;
3. Compute ~u = ~x + σ ~en;
2 4. Compute π = σ + σξn;
5. Obtain I − π−1uuT .
The matlab code for computing U is as follows:
2 Yingwei Wang Numerical Linear Algebra function U=ref_n(x) %%% The elementary reflector for nth place %%% x: column vector %%% U: matrix such that Ux = -\simga e_n.
%% preprare n=length(x); %% compute \sigma s = sign(x(n))*norm(x); %% compute \pi p = s^2 + s*x(n); %% compute u e = eye(n); u = x + s*e(:,n); %% compute U U = e - (u*u’)/p;
A numerical test is given here.
>> x = [1 2 3]’; >> U=ref_n(x)
U =
0.9604 -0.0793 -0.2673 -0.0793 0.8414 -0.5345 -0.2673 -0.5345 -0.8018
>> U*x ans =
-0.0000 -0.0000 -3.7417
3 Yingwei Wang Numerical Linear Algebra
3 QR decomposition for a matrix with special form
Question: Let A have the form R A = , (3.1) S where R is upper triangular. Describe an algorithm that has elementary reflectors to reduce A to upper trapezoidal form taking advantage of the zero elements in R. Solution: Here is the Matlab code. function B=RS(R,S) %%% QR decomposition for A=[R;S], %%% R = upper triangluar matrix; %%% S = full matrix; %%% B = upper trapezoidal form of A;
%% prepare nr = size(R); ns = size(S); n = nr(2); m = nr(1)+ns(1); m2 = ns(1);
%% condition if m %% main loop P = eye(m); for i=1:n % choose the nonzero vector x = [R(i,i);S(:,i)]; % compute the elementary reflector U=ref_1(x); % update R and S rs = [R(i,i:end);S(:,i:end)]; rs = U*rs; R(i,i:end) = rs(1,:); S(:,i:end) = rs(2:end,:); 4 Yingwei Wang Numerical Linear Algebra end B = [R;S]; Here is a numerical test: >> R = eye(3) R = 1 0 0 0 1 0 0 0 1 >> S = ones(3) S = 1 1 1 1 1 1 1 1 1 >> B=RS(R,S) B = -2.0000 -1.5000 -1.5000 0 -1.3229 -0.5669 0 0 -1.1952 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0 0 If we use the Matlab function qr.m, we can get the same result [Q1,R1] = qr(A) Q1 = -0.5000 0.5669 0.3586 -0.3162 -0.3162 -0.3162 5 Yingwei Wang Numerical Linear Algebra 0 -0.7559 0.3586 -0.3162 -0.3162 -0.3162 0 0 -0.8367 -0.3162 -0.3162 -0.3162 -0.5000 -0.1890 -0.1195 0.7721 -0.2279 -0.2279 -0.5000 -0.1890 -0.1195 -0.2279 0.7721 -0.2279 -0.5000 -0.1890 -0.1195 -0.2279 -0.2279 0.7721 R1 = -2.0000 -1.5000 -1.5000 0 -1.3229 -0.5669 0 0 -1.1952 0 0 0 0 0 0 0 0 0 4 Minimizing problem with constraint Question: Let A ∈ Rm×n have linearly independent columns and C ∈ Rl×n have linearly independent rows. Let d ∈ Rl. Derive an algorithm for solving the least squares problem with constraint : min kb − Axk2 (4.1) x Cx = d. (4.2) Solution: Algorithm is as follows: 1. Find orthogonal U ∈ Rm×m so that R UA = , 0 where R ∈ Rn×n is upper triangular. Similarly, find orthogonal V ∈ Rn×n so that CV =(L, 0) where L ∈ Rl×l is lower triangular. 6 Yingwei Wang Numerical Linear Algebra Let y = V T x, then x = Vy and the problem (4.1)-(4.2) becomes min kUb − UAVyk2 (4.3) y CVy = d. (4.4) 2. Let us start from Eq.(4.4). CVy = d, y ⇒ (L, 0) 1 = d, y2 ⇒ Ly1 = d, l n−l where y1 ∈ R and y2 ∈ R . Now we can solve for y1. 3. Let y1 x = Vy =(V1,V2) = V1y1 + V2y2, y2 n×l n×(n−l) where V1 ∈ R and V2 ∈ R . Then by (4.3), we can get R b − Rx Ub − UAx = Ub − x = 1 0 b2 ⇒ b1 − Rx = b1 − R(V1y1 + V2y2)=(b1 − RV1y1) − RV2y2. Since y1 is known, now the problem (4.3) with constraint (4.4) becomes the following problem without of constraint: min k(b1 − RV1y1) − RV2y2k, (4.5) y2 n n×(n−l) where (b1 − RV1y1) ∈ R and RV2 ∈ R . 4. Let ˜b = b1 − RV1y1 and A˜ = RV2, then (4.5) becomes min k˜b − Ay˜ 2k, (4.6) y2 which can be solved by regular method shown in the class. For instance, we can do QR decomposition on A˜. 7 Yingwei Wang Numerical Linear Algebra 5 Plane rotation Question: Let A ∈ Rm×n. Use plane rotations to introduce zeros into A in the order illustrated below: x x x x1 x x x2 x5 x , (5.1) x3 x6 x8 x4 x7 x9 without disturbing the previously introduced zeros. Give a detailed algorithm and a Matlab code, for this reduction to upper trapezoidal form and test your code on a random 5 × 3 matrix of rank 3. Solution: 5.1 Algorithm The algorithm is as follows: 1. Given A =(aij)m×n. 2. Let P = Im, do the main loop: for i=1:n for j=(i+1): m 2 2 x1 = A(i, i), x2 = A(j, i), η = x1 + x2, x x q c = 1 , s = 2 , η η pp = Im, pp(i, i)= c, pp(j, j)= c, pp(i, j)= s, pp(j, i)= −s, P = pp ∗ P, A = pp ∗ A, endfor endfor Now, A becomes upper trapezoidal and P is the transform matrix. 8 Yingwei Wang Numerical Linear Algebra 5.2 Matlab code The Matlab code is as follows: function [P,B]=plane_rot(A) %%% A is m by n and m>n; %%% B is upper trapezoidal %%% P is plane rotation such that PA=B. %% prepare [m,n]=size(A); P = eye(m); %% main loop for i=1:n for j = (i+1):m x1 = A(i,i); x2 = A(j,i); et = sqrt(x1^2 + x2^2); c = x1/et; s = x2/et; % update P pp = eye(m); pp(i,i) = c; pp(j,j) = c; pp(i,j) = s; pp(j,i) = -s; P = pp*P; % update A A = pp*A end end B=A; 5.3 Numerical test The numerical test and results are shown here. >> A = [magic(3);ones(2,3)] 9 Yingwei Wang Numerical Linear Algebra A = 8 1 6 3 5 7 4 9 2 1 1 1 1 1 1 >> [P,B]=plane_rot(A) A = 8.5440 2.6919 8.0758 0 4.3305 4.4476 4.0000 9.0000 2.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 A = 9.4340 6.2540 8.1620 0 4.3305 4.4476 -0.0000 7.0096 -1.6128 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 A = 9.4868 6.3246 8.2219 0 4.3305 4.4476 -0.0000 7.0096 -1.6128 0 0.3352 0.1341 1.0000 1.0000 1.0000 A = 10 Yingwei Wang Numerical Linear Algebra 9.5394 6.3945 8.2815 0 4.3305 4.4476 -0.0000 7.0096 -1.6128 0 0.3352 0.1341 0 0.3315 0.1326 A = 9.5394 6.3945 8.2815 -0.0000 8.2394 0.9655 -0.0000 0 -4.6314 0 0.3352 0.1341 0 0.3315 0.1326 A = 9.5394 6.3945 8.2815 -0.0000 8.2462 0.9701 -0.0000 0 -4.6314 0.0000 -0.0000 0.0947 0 0.3315 0.1326 A = 9.5394 6.3945 8.2815 -0.0000 8.2529 0.9747 -0.0000 0 -4.6314 0.0000 -0.0000 0.0947 0.0000 0 0.0935 A = 9.5394 6.3945 8.2815 11 Yingwei Wang Numerical Linear Algebra -0.0000 8.2529 0.9747 0.0000 -0.0000 4.6324 -0.0000 0.0000 0.0000 0.0000 0 0.0935 A = 9.5394 6.3945 8.2815 -0.0000 8.2529 0.9747 0.0000 -0.0000 4.6333 -0.0000 0.0000 0.0000 0.0000 0.0000 0 P = 0.8386 0.3145 0.4193 0.1048 0.1048 -0.5286 0.3622 0.7656 0.0399 0.0399 -0.0928 0.8725 -0.4789 0.0201 0.0201 0.0662 0.0662 0.0662 -0.9934 0 -0.0654 -0.0654 -0.0654 -0.0131 0.9935 B = 9.5394 6.3945 8.2815 -0.0000 8.2529 0.9747 0.0000 -0.0000 4.6333 -0.0000 0.0000 0.0000 0.0000 0.0000 0 We can see that we introduce 0 into A in the order we want. 12 CS 515: Homework 5 Yingwei Wang ∗ Department of Mathematics, Purdue University, West Lafayette, IN, USA 1 Basic iterative methods Question: Write Matlab programs for the Jacobi, Gauss-Seidel, and symmetric Gauss-Seidel iterative schemes for solving linear systems of the form Ax = f. Start with an initial iterate x0 = 0, and stopping criterion r k kk2 10−3. r ≤ k 0k Solution: 1.1 Matlab code My matlab codes for Jacobi, Gauss-Seidel, and symmetric Gauss-Seidel iterative schemes are: function [x,N,er]=jacobi(a,b) %%% Use Jacobi iteration to solve Ax=b, %%% x0 is initial value %% tolerance ep = 1e-3; %% matrices D=diag(diag(a)); E=inv(D); %% main loop ∗E-mail address: [email protected]; Tel: 765 237 7149 1 Yingwei Wang Numerical Linear Algebra x0 = zeros(size(b)); r = b - a*x0; r0 = norm(r); i=0; rr=[]; NN = []; nr = r0; while nr>=ep & ep<50 d = E*r; x0 = x0 + d; r = b - a*x0; NN = [NN;i]; nr = norm(r)/r0; rr = [rr;nr]; i=i+1; end x= x0; N = i; er = norm(r)/nr; %% plot plot(NN,rr,’*-’) title(’Jacobi iteration’) xlabel(’The number of iterations’); ylabel(’Relative residues’) function [x,N,er]=gs(a,b) %%% Use Gauss-seidel iteration to solve Ax=b, %%% x0 is initial value %% tolerance ep = 1e-3; %% matrices D = diag(diag(a)); L = -tril(a,-1); E = inv(D-L); %% main loop 2 Yingwei Wang Numerical Linear Algebra x0 = zeros(size(b)); r = b - a*x0; r0 = norm(r); i=0; rr=[]; NN = []; nr = r0; while nr>=ep & i< 50 d = E*r; x0 = x0 + d; r = b - a*x0; NN = [NN;i]; nr = norm(r)/r0; rr = [rr;nr]; i=i+1; end x= x0; N = i; er = norm(r)/nr; %% plot plot(NN,rr,’*-’) title(’Gauss-Seidel iteration’) xlabel(’The number of iterations’); ylabel(’Relative residues’) function [x,N,er]=sgs(a,b) %%% Use symmetric Gauss-seidel iteration to solve Ax=b, %%% x0 is initial value %% tolerance ep = 1e-3; %% matrices D = diag(diag(a)); L = -tril(a,-1); U = - triu(a,1); M1 = inv(D-L); M2 = inv(D-U); 3 Yingwei Wang Numerical Linear Algebra H1 = M1*U; g1 = M1*b; H2 = M2*L; g2 = M2*b; %% main loop x0 = zeros(size(b)); r = b - a*x0; r0 = norm(r); i=0; rr=[]; NN = []; nr = r0; while nr>=ep & i <50 xt = H1*x0 + g1; x0 = H2*xt + g2; r = b - a*x0; NN = [NN;i]; nr = norm(r)/r0; rr = [rr;nr]; i=i+1; end x= x0; N = i; er = norm(r)/nr; %% plot plot(NN,rr,’*-’) title(’Symmetric Gauss-Seidel iteration’) xlabel(’The number of iterations’); ylabel(’Relative residues’) 4 Yingwei Wang Numerical Linear Algebra 1.2 Test Test all three programs for solving the system given by 1101 0 0 0 2 0202 0 0 0 3 0 0 30 0 A = , (1.1) 121 0 0 0 1 − 0 0 0 2 2 24 − 0 0 1 12 1 0 − f T , , , , , krkk2 and = (1 2 3 1 2 1), and plot the relative residual kr0k2 v.s. the number of itertations. First, let us do the row permutations to matrix A in order to get a diagonal dominant matrix. Let P =(e4, e1, e2, e6, e3, e5), then 121 0 0 0 1 − 1101 0 0 0 2 0202 0 0 P A = , (1.2) 0 0 1 12 1 0 − 0 3 0 0 30 3 0 0 0 2 2 24 − 1 1 2 P f = . (1.3) 1 3 2 Now we can use the Matlab codes to solve the linear system (P A)x = (P f). The results are shown in Figs. 1-3, which indicate that the convergence rate of Jacobi, Gauss-Seidel (GS) and symmetric Gauss-Seidel (SGS) iterative are SGS > GS > Jacobi. 5 Yingwei Wang Numerical Linear Algebra Jacobi iteration 0 10 −1 10 −2 10 Relative residues log(error) −3 10 −4 10 0 0.5 1 1.5 2 2.5 3 The number of iterations N Figure 1: Jacobi iteration Gauss−Seidel iteration −1 10 −2 10 −3 10 Relative residues log(error) −4 10 0 0.5 1 1.5 2 The number of iterations N Figure 2: Gauss-Seidel iteration 2 Matrices with property A Definition 2.1. A matrix A is said to have property A if it can be permutated into the form D E Aˆ = P AP T = 1 , (2.1) FD2 where D1 and D2 are diagonal matrices, and P is a permutation matrix. 6 Yingwei Wang Numerical Linear Algebra Symmetric Gauss−Seidel iteration −2 10 −3 10 −4 10 Relative residues log(error) −5 10 0 0.2 0.4 0.6 0.8 1 The number of iterations N Figure 3: Symmetric Gauss-Seidel iteration 2.1 (a) Show that the following matrix has property A: 4 1 − 1 4 1 − − A = ∗ ∗ ∗ . ∗ ∗ ∗ 1 4 1 − − 1 4 − n×n What are D1,D2,E,F and P . Solution: The answer depends on the fact that n is odd or even. 1. n =2k. P = [e , e , , e , e , e , , e ], 1 3 ··· 2k−1 2 4 ··· 2k D1 = D2 =4Ik, 1 − 1 1 E = − − , ∗ ∗ 1 1 − − k×k F = ET . 7 Yingwei Wang Numerical Linear Algebra 2. n =2k + 1. P = [e , e , , e , e , e , , e ], 1 3 ··· 2k+1 2 4 ··· 2k D1 =4Ik+1, D2 =4Ik, 1 − 1 1 − − E = , ∗ ∗ 1 1 − − 1 − (k+1)×k F = ET . 2.2 (b) Using form (2.1), determine whether Jacobi, Gauss-Seidel, and symmetric Gauss-Seidel iter- ative schemes converge, and if so, what are the respective asymptotic rates of convergence? Solution: Let Aˆ = D L LT , where − − D =4In, 0 0 L = . F 0 − 1. Jacobi iteration. T In this case, MJ = D, NJ =(L + L ), then −1 −1 0 D1 E HJ = M NJ = −1 − . J D F 0 − 1 Consider the eigenvalue problem HJ v = λJ v, −1 0 D1 E v1 v1 −1 − = λJ ⇒ D F 0 v2 v2 − 2 (D−1F )(D−1E)v = λ v2, ⇒ 2 1 2 J 2 1 F Ev = λ v2, ⇒ 16 2 J 2 1 Iv = λ v2, ⇒ 16 2 J 2 1 λ = . ⇒ J 4 8 Yingwei Wang Numerical Linear Algebra 2. Gauss-Seidel iteration. In this case, M = D L, N = LT , then GS − GS −1 −1 0 D1 E HGS = MGS NGS = −1 −1 . 0 D2 FD1 E Consider the eigenvalue problem HGSv = λGSv, we can get 1 max(λ )= λ2 = . GS J 16 3. Symmetric Gauss-Seidel iteration. ˆ −1/2 −1/2 1 Let L = D LD = 4 L, then the iteration matrix is H =(I Lˆ)−1LˆLˆT (I Lˆ)−T , SGS − − I 0 1 0 0 I 1 E = − 4 1 F I 16 E F 0 I − 4 1 0 0 = 16 E F Now we know that 1 max(λ )= . SGS 16 3 (c) Let n = 1024, f T = (1, 0, , 0, 1). Test all three iterative schemes on form (2.1). The results ··· are shown in Figs. 4-6. All of them are convergent. Besides, the rate of convergence is rGS =2rJ ,rSGS =2rGS. 9 Yingwei Wang Numerical Linear Algebra Jacobi iteration 0 10 −1 10 −2 10 Relative residues log(error) −3 10 −4 10 0 1 2 3 4 5 6 7 The number of iterations N Figure 4: Jacobi iteration Gauss−Seidel iteration 0 10 −1 10 −2 10 Relative residues log(error) −3 10 −4 10 0 1 2 3 4 5 The number of iterations N Figure 5: Gauss-Seidel iteration 4 SOR iteration Solve the system Ax = f, where 1 a A = − . a 1 using the iterative system 1 0 1 ω aω xk+1 = − xk + ωf. ωa 1 0 1 ω − − 10 Yingwei Wang Numerical Linear Algebra Symmetric Gauss−Seidel iteration −1 10 −2 10 −3 10 Relative residues log(error) −4 10 0 0.5 1 1.5 2 The number of iterations N Figure 6: Symmetric Gauss-Seidel iteration 4.1 (a) For which values of a is the scheme convergent for ω = 1. Solution: For ω = 1, we have 1 0 0 a M = , N = . a 1 0 0 − Then the iteration matrix is 1 0 0 a 0 a H = M −1N = = . a 10 0 0 a2 In order to get a convergent scheme, we have ρ(H)= a2 < 1 a < 1. ⇒| | 4.2 (b) Let a =0.5. Find the value of ω which minimizes the spectral radius of the iteration matrix. Solution: Let A = D L U, where − − 0 0 0 a D = I, L = , U = . a 0 0 0 11 Yingwei Wang Numerical Linear Algebra Now the iteration matrix can be written as H =(D ωL)−1[(1 ω)D + ωU], − − λI H =(I ωD−1L)−1[(λ + ω 1)I ωD−1(λL + U)], ⇒ − − − − det(λI H)=(λ + ω 1)2 λω2a2. ⇒ − − − Let det(λI H) = 0, we can get the equation − λ + ω 1= λ1/2ωa, (4.1) − ± 1 where a = 2 Let λ + ω 1 y (λ)= − , 1 ω 1 y (λ)= λ1/2. 2 2 In order to get the minimal λ, we should choose ω such that y1(λ)= y2(λ) ′ ′ , y1(λ)= y2(λ) λ+ω−1 1 1/2 ω = 2 λ 1 1 , ⇒ ω = 4λ1/2 1 ω2 ω +1=0, ⇒ 16 − ω =8 4√3 1.0718, ⇒ − ≈ since ω (0, 2). ∈ 12 CS 515: Homework 6 Yingwei Wang ∗ Department of Mathematics, Purdue University, West Lafayette, IN, USA 1 Problem 1, page 145 Consider the linear system Ax = b, where A is s.p.d. 1.1 (a) Question: Consider the sequence of one dimensional projection processes with K = L = span {ei}, where the sequence of indices i is selected in any fashion. Let xnew be a new iteration after one projection step from x and let r = b − Ax, d = A−1b − x, −1 dnew = A b − xnew. Show that 2 (r, ei) (Adnew,dnew)=(Ad,d) − . (1.1) aii Proof. We know that T −1 T xnew = x + ei(ei Aei) ei r, (r, ei) = x + ei. aii ∗E-mail address: [email protected]; Tel: 765 237 7149 1 Yingwei Wang Numerical Linear Algebra Then, we have −1 −1 (r, ei) (r, ei) dnew = A b − xnew = A b − x − ei = d − ei, aii aii (r, ei) (r, ei) Adnew = b − Ax − ei = r − Aei. aii aii Here we use Ad = r. Now we can get (r, ei) (r, ei) (Adnew,dnew) = r − Aei,d − ei , aii aii 2 2 (r, ei) (r, ei) (r, ei) =(Ad,d) − − (d, Aei) + 2 (Aei, ei), aii aii aii 2 2 (r, ei) (r, ei) (r, ei) =(Ad,d) − − (Ad, ei) + 2 aii, aii aii aii (r, e )2 (r, e )2 (r, e )2 =(Ad,d) − i − i + i , aii aii aii (r, e )2 =(Ad,d) − i . aii If (r, ei) = 0, then we can get kdnewkA = kdkA. It follows that the algorithm may not be convergent. 1.2 (b) Question: Assume that i is selected at each projection step to be the index of a component of largest absolute value in the current residual vector r = b − Ax. Show that 1 1/2 kd k ≤ 1 − kdk , (1.2) new A nκ2(A) A in which κ(A) is the spectral condition number of A. Proof. Since i is selected at each projection step to be the index of a component of largest absolute value in r, we know that −1/2 |(r, ei)| ≥ n krk2, krk2 ⇒ |(r, e )|2 ≥ 2 . i n 2 Yingwei Wang Numerical Linear Algebra Besides, since aii ≤kAk2, we know that (r, e )2 krk2 i ≥ 2 , aii nkAk2 krk kA−1k krk 2 2 2 , = −1 n kAk2 kA k2 krk kA−1rk ≥ 2 2 , n κ2(A) (r, A−1r) ≥ , n κ2(A) (Ad,d) = , n κ2(A) kdk = A . (1.3) n κ2(A) By (1.1) and (1.3), we can get (1.2). 1/2 1 Since 1 − nκ2(A) < 1, we know that the algorithm is convergent. 2 Problem 4, page 146 Consider the matrix 1 −6 0 A = 6 2 3 . 0 3 2 2.1 (a) Question: Find a rectangle or square in the complex plane that contains all of the eigen- values of A. Solution: By the Gerschgorin theorem, we know that all of the eigenvalues of the matrix A lie in this square S = {(x, y) | −7 2.2 (b) Question: Is the Minimal Residual iteration guaranteed to converge for a linear system with the matrix A? 3 Yingwei Wang Numerical Linear Algebra Solution: Choose x = (0, 1, −2)T , then xT Ax = −2 < 0. It implies that A is not positive definite. So we do not know whether the MR iteration is convergent or not. 3 Problem 7, page 147 Consider the projection method that update at each step the current solution with linear combinations from two directions: the current residual r and Ar. 3.1 (a) Question: Consider an orthogonal projection method; i.e. at each step L = K = span {r, Ar}. Assuming that A is s.p.d, establish convergence of the algorithm. Solution: We know that xk+1 = xk + αrk + βArk, ⇒ rk+1 = rk − A(αrk + βArk) ⊥ [rk, Ark], 2 (rk − αArk − βA rk,rk)=0, ⇒ 2 (rk − αArk − βA rk, Ark)=0, 2 (rk,rk)= α(Ark,rk)+ β(A rk,rk), ⇒ 2 (rk, Ark)= α(Ark, Ark)+ β(A rk, Ark), 2 (rk,rk)= α(Ark,rk)+ β(A rk,rk), ⇒ 2 3 (Ark,rk)= α(A rk,rk)+ β(A rk,rk), Let x∗ be the exact solution of Ax = b, and d = x∗ − xk be the error. Let us consider 4 Yingwei Wang Numerical Linear Algebra the A-norm of the error. 2 kdk+1kA =(Adk+1,dk+1), =(rk+1,dk+1), =(rk+1,dk − αrk − βArk), =(rk+1,dk), 2 =(rk − αArk − βA rk,dk), 2 =(rk,dk) − α(rk, Adk) − β(A rk,dk), =(Adk,dk) − α(rk,rk) − β(Ark,rk), 2 = kdkkA − αkrkk − βkrkkA. It follows that 2 2 kdk+1kA ≤kdkkA. 3.2 (b) Question: Consider a least-squares projection method; i.e. at each step K = span {r, Ar}, L = AK. Assuming that A is positive definite, establish convergence of the algorithm. Solution: We know that xk+1 = xk + αrk + βArk, 2 ⇒ rk+1 = rk − A(αrk + βArk) ⊥ [Ark, A rk], 2 (rk − αArk − βA rk, Ark)=0, ⇒ 2 2 (rk − αArk − βA rk, A rk)=0, 2 2 (rk, Ark)= α(Ark, Ark)+ β(A rk, A rk), ⇒ 2 2 2 2 (rk, A rk)= α(Ark, A rk)+ β(A rk, A rk), 2 Let us consider the residual rk = b − Axk, rk+1 = rk − αArk − βA rk. 2 2 krk+1k2 =(rk − αArk − βA rk,rk), 2 =(rk,rk) − α(Ark,rk) − β(A rk,rk), 2 2 2 = krkk2 − αkrkkA − βkrkkA2 . It implies that 2 2 krk+1k2 ≤krkk2. 5 Yingwei Wang Numerical Linear Algebra 4 Problem 13, page 148 Consider the following real-valued function of the vector variable x, where A and b are the −1 coefficient matrix and right hand system of a given linear system Ax = b and x∗ = A b. 2 a(x)= kx∗ − xk2, 2 f(x)= kb − Axk2, T T 2 g(x)= kA b − A Axk2, h(x)=2(b, x) − (Ax, x). 4.1 (a) Question: Calculate the gradient of all four functions above. Solution: 2 T T a(x)= kx∗ − xk2 =(x∗ − x )(x∗ − x), T T ⇒ ∇a(x)= −2(x∗ − x ). 2 T T T f(x)= kb − Axk2 =(b − x A )(b − Ax), ⇒ ∇f(x)= −2(bT − xT AT )A. T T 2 T T T T T g(x)= kA b − A Axk2 =(b A − x A A)(A b − A Ax), ⇒ ∇g(x)= −2(bT A − xT AT A)AT A = −2(bT − xT AT )AAT A. h(x)=2(b, x) − (Ax, x)=2bT x − xT AT x, ⇒ ∇h(x)=2bT − xT (AT + A). 4.2 (b) Question: How is the gradient of g related to that of f? Solution: It is obvious that ∇g(x)=(∇f(x))AT A. 4.3 (c) Question: How is the gradient of f related to that of h when A is symmetric? Solution: It is obvious that ∇f(x)= −(∇h(x))A. 6 Yingwei Wang Numerical Linear Algebra 4.4 (d) Question: How does the function h relate to the A-norm of the error x∗ − x when A is s.p.d.? Solution: We know that −1 kx∗ − xkA = kA b − xkA, =(b − Ax, A−1b − x), =(b, A−1b) − (b, x) − (Ax, A−1b)+(Ax, x), = kbkA−1 − 2(b, x)+(Ax, x), = kbkA−1 − h(x). 5 Steep descent Solve the system Ax = f, where 4 −1 3 −1 4 −1 2 ∗ ∗ ∗ A = , f = ··· . ∗ ∗ ∗ 2 −1 4 −1 3 −1 4 n×1 n×n 5.1 Matlab code Since A is s.p.d, we should use the steepest descent iteration. My matlab code for steepest descent iteration is function sd %%% Steepest descent algorithm %% linear system n = 1024; A = 4*diag(ones(n,1)) - diag(ones(n-1,1),-1) - diag(ones(n-1,1),1); f = 2*ones(n,1); f(1)=3; f(end)=3; %% exact solution 7 Yingwei Wang Numerical Linear Algebra y = ones(n,1); %% main loop ep = (1e-4)*norm(f); x = zeros(n,1); er=1; i=0; dd = []; ii = []; while er>ep r = f - A*x; c = (r’)*r/((r’)*A*r); x = x + c*r; i=i+1; er = norm(r); ii = [ii,i]; d = (r’)*(y-x); dd = [dd,d]; end x; %% plot A norm of the errors semilogy(ii,dd,’*-’) xlabel(’the number of iterations’) ylabel(’log(A-norm of the errors)’) title(’Steepest descent’) 5.2 Numerical results n x krkk2 −4 Run the programme for = 1024 and 0 = 0, with stopping criterion kr0k2 ≤ 10 . The A-norm of the errors vs. the number of the iterations are shown in Fig.1. 8 Yingwei Wang Numerical Linear Algebra Steepest descent 1 10 0 10 −1 10 −2 10 −3 10 −4 10 log(A−norm of the errors) −5 10 −6 10 −7 10 1 2 3 4 5 6 7 8 9 the number of iterations Figure 1: A-norm of the errors vs. the number of the iterations in Steepest Descent 6 Minimal residual Solve the system By = g, where 2 −1 1 1 2 −1 2 ∗ ∗ ∗ B = , g = ··· . ∗ ∗∗ 2 1 2 −1 3 1 2 n×1 n×n 6.1 Matlab code Since B is nonsingular, we should use the minimal residual iteration. My matlab code for minimal residual iteration is function mr %%% Minimal residual algorithm %% linear system 9 Yingwei Wang Numerical Linear Algebra n = 1024; A = 2*diag(ones(n,1)) + diag(ones(n-1,1),-1) - diag(ones(n-1,1),1); f = 2*ones(n,1); f(1)=1; f(end)=3; %% exact solution y = ones(n,1); %% main loop f1 = norm(f); ep = (1e-3)*f1; x = zeros(n,1); er=1; i=0; dd = []; ii = []; while er>ep r = f - A*x; c = ((r’)*A*r)/((r’)*(A’)*A*r); x = x + c*r; i=i+1; er = norm(r); ii = [ii,i]; d = er/f1; dd = [dd,d]; end x; %% plot A norm of the errors semilogy(ii,dd,’*-’) xlabel(’the number of iterations’) ylabel(’log(Relative residual)’) title(’ Minimal Residual’) 10 Yingwei Wang Numerical Linear Algebra 6.2 Numerical results n y krkk2 −3 Run the programme for = 1024 and 0 = 0, with stopping criterion kr0k2 ≤ 10 . The relative residuals vs. the number of the iterations are shown in Fig.2. Minimal Residual 0 10 −1 10 −2 10 log(Relative residual) −3 10 −4 10 1 2 3 4 5 6 7 8 9 the number of iterations Figure 2: relative residuals vs. the number of the iterations in Minimal Residual 11