Math 408 Advanced Linear Algebra Chi-Kwong Li Chapter 4 Hermitian

Math 408 Advanced Linear Algebra Chi-Kwong Li

Chapter 4 Hermitian and symmetric matrices

Basic properties

Theorem Let A ∈ Mn. The following are equivalent. (a) A is Hermitian, i.e., A = A∗. (b) x∗Ax ∈ R for all x ∈ Cn. (c) A is normal with real eigenvalues.

Remark ∗ ∗ (a) Let A ∈ Mn. Then A = H + iG, where H = (A + A )/2 and G = (A − A )/(2i) are Hermitian. Also, AA∗ and A∗A are Hermitian. If A is Hermitian, then Ak is Hermitian for positive integer k, and A−1 is Hermitian if A is invertible. (b) The set of Hermitian matrices form a real linear space. (c) The product of Hermitian matrices may not be Hermitian. (d) The product of two Hermitian matrices is Hermitian if and only if they commute. (e) A matrix A is the product of two Hermitian matrices if and only if A is similar to A∗ (via a Hermitian matrix), equivalently, A is similar to a real matrix. Proof. (a) If A is Hermitian, then A = U ∗DU for some real diagonal matrix D. So, Ak = U ∗DkU is Hermitian for positive integers k (or negative integers if A−1 exists).

(b) Let Hn be the set of n × n Hermitian matrices. Then 0 ∈ Hn, and rA + sB ∈ Hn for any

A, B ∈ Hn and r, s ∈ R.

(c) Consider A = E12 + E21 and B = E11 − E22. ∗ ∗ (d) If A, B ∈ Hn commute then A = U D1U and B = U D2U for some unitary U and real ∗ ∗ diagonal D1,D2. Thus, AB = U (D1D2)U is unitary. If A, B, AB ∈ Hn, then AB = (AB) = B∗A∗ = BA. ∗ (e) Suppose A = HK with H,K ∈ Hn. If H = U DU for some unitary U and D = D1 ⊕0n−k, D K ∗ where D is an invertible real diagonal matrix in H . Then U ∗AU = DU ∗KU = 1 11 1 k 0 0 K D 0 and U ∗A∗U = 11 1 . Now, K D and DK have the same Jordan blocks for the nonzero ∗ 0 11 1 11 eigenvalues, and rank (Am) = rank ((A∗)m) for all m. Thus, A and A∗ have the same Jordan form and are similar. Suppose AS = SA∗ with S = H+iG. Then S∗A∗ = AS∗ implies that A(H+iG) = (H+iG)A∗ and (H − iG)A∗ = A(H − iG). Hence, AH = HA∗ and AG = GA∗. Hence AK = KA∗ for some invertible K = H + tG. Thus, A = K(A∗K−1), where (A∗K−1)∗ = K−1A = A∗K−1. Suppose A and A∗ are similar. Then they have the same Jordan blocks. Thus, the Jordan blocks Jm(λ) and Jm(λ) will occur together in A for any nonreal eigenvalue λ, and can be combined to a real Jordan block. Thus, A is similar to a direct sum of real Jordan blocks.

1 Suppose S−1AS = R for a real matrix R. Then S−1AS = R = T −1RtT = T −1R∗T = T −1S∗A∗(S−1)∗T . Thus, A and A∗ are similar.

Eigenvalue inequalities

Denote by λ1(A) ≥ · · · ≥ λn(A) the eigenvalues of a Hermitian matrix A ∈ Mn.

Theorem (Courant-Fischer-Weyl) Suppose A ∈ Mn is Hermitian. Then for k ∈ {1, . . . , n}.

∗ ∗ λk(A) = max min v Av = min min v Av. dim W =k v∈W,v∗v=1 dim W =n−k+1 v∈W,v∗v=1

Proof. For simplicity, write λj = λj(A) for all j. Suppose {u1, . . . , un} is a set of orthonormal eigenvectors of A such that Auj = λjuj. For any subspace of dimension W , say, spanned by Pn w1, . . . , wk, it contains a unit vector v in the span of {uk, . . . , un}. Thus, v = j=k µjuj with Pn 2 j=k |µj| = 1 such that

n n ∗ X 2 X 2 v Av = |µj| λj ≤ |µj| λk = λk. j=k j=k

Pk Let W be the span of u1, . . . , uk. For any v ∈ W , we have v = j=1 γjvj so that

k n ∗ X 2 X 2 v Av = |γj| λj ≥ |γj| λk = λk j=1 j=k and the equality holds if v = uk. The ﬁrst equality holds. The proof is complete.

n Corollary (Rayleigh principle) Let A ∈ Mn be Hermitian. For any unit vector x ∈ C ,

∗ λ1(A) ≥ v Av ≥ λn(A).

Corollary (Interlacing inequalities) Let A ∈ Mn be Hermitian with eigenvalues a1 ≥ · · · ≥ an and

B ∈ Mn−1 be a principal submatrix of A with eigenvalues b1 ≥ · · · ≥ bn−1. Then

a1 ≥ b1 ≥ a2 ≥ · · · ≥ an−1 ≥ bn−1 ≥ an.

Theorem (Fan-Pall) Suppose a1, . . . , an and b1, . . . , bn−1 satisfy the interlacing inequalities. Then there is a Hermitian matrix A ∈ Mn with a principal submatrix B ∈ Mn−1 with a1, . . . , an and b1, . . . , bn−1 as eigenvalues.

Proof. By the discussion in class, We may assume that a1 > b1 > a2 ··· > an−1 > bn−1 > bn.

Let D = diag (a1, . . . , an). We show that there is a unitary U such that if we delete the ﬁrst row and ∗ ﬁrst column of the matrix A = UDU , we get a matrix B ∈ Hn−1 with eigenvalues b1 ≥ · · · ≥ bn. ∗ To see this, note that for t∈ / {a1, . . . , an}, we have tI − A = U(tI − D)U is invertible with inverse

(tI − A)−1 = U(t − D)−1U ∗. (1)

2 Recall that the (1, 1) entry of an invertible matrix C equals det(C(1, 1))/ det(C) using the adjoint matrix formula, where C(1, 1) is obtained from C by removing its first row and first column. So, the (1, 1) entry of (tI − A)−1 equals det(tI − B)/ det(tI − A) as B is the matrix obtained from A by removing the first row and the first column.

Now, consider the right side of (1). If the ﬁrst row of U equals (x1, . . . , xn), then the (1, 1) entry of the matrix is |x |2 |x |2 f(t) = 1 + ··· + n . t − a1 t − an

Equating the two expressions for the (1, 1) entry of (tI − A)−1 mentioned above, we have

n 2 n X |xj| X 2 Y det(tI − B) = det(tI − A) = |xj| (t − ak). t − aj j=1 j=1 k6=j

Qn−1 Q Homework 1. (a) Show that for all i = 1, . . . , n, the value wi = j=1 (ai − bj)/ j6=i(ai − aj) is positive.

By our assumption that a1 > b1 > ··· bn−1 > bn, we see that for j 6= i, we have ai > bj if and only if ai > aj. Thus, the numerator and the denominators have the same number of negative terms. Thus, the quotient is positive. √ (b) If we choose xi = wi for all i = 1, . . . , n, show that

n 2 n−1 X |xj| Y det(tI − A) = (t − b ) t − a j j=1 j j=1 for all t = a1, . . . , an, and hence the two polynomials in t are identical.

If we substitute ai on the left hand side, all the summands are zero except for the one involving 2 2 Qn−1 |xi| , and by the deﬁnition of |xi| the expression reduces to j=1 (ai − bj). Since both side are polynomials of degree n − 1, we see that the two polynomail are identical.

(c) Show that if we choose a unitary U with the ﬁrst row x = (x1, . . . , xn) satisfying (b), then ∗ U AU has a principal submatrix with eigenvalues b1 ≥ · · · ≥ bn−1 as asserted. By part (a) and (b), and the previous discussion, such a choice of U will yield a submatrix B ∗ Qn−1 of A = UDU such that det(tI − B) = j=1 (t − bj). Thus, B has the desired eigenvalues.

Deﬁnition Let c = (c1, . . . , cn) and d = (d1, . . . , dn) be real vectors. We say that c is majorized by d, denoted by c ≺ d if the sum of the k largest entries of c is not larger than that of d for k = {1, . . . , n − 1}, and the sum of all entries of c is the same as that of d.

Theorem Suppose a = (a1, . . . , an) and d = (d1, . . . , dn) are real vectors. Then there is a Her- mitian matrix in Mn with eigenvalues a1, . . . , an and diagonal entries d1, . . . , dn if and only if d ≺ a.

3 2 Homework 2. (a) Suppose n = 2. If (d1, d2) ≺ (a1, a2), show that there is a vector v ∈ R such t that v Dv = d1, where D = diag (a1, a2). Furthermore, if P is an orthogonal matrix with the ﬁrst t column equal to v, then P DP has diagonal entries d1, d2. t t 2 2 Note that a2 ≤ d1 ≤ a1. Assume v = (cos θ, sin θ). Then f(θ) = v Dv = a1 cos θ + a2 sin θ.

Now, f(0) = a1 ≥ d1 and f(π/2) = a2 ≤ d1. By continuity, there is θ ∈ [0, π/2] such that f(θ) = d − 1. t t Let w = (− sin θ, cos θ). Then P = [v|w] is orthogonal, P DP has (1, 1) entry d1. By the t trace condition, we see that the (2, 2) entry of P DP is d2 so that a1 + a2 = d1 + d2.

(b) Suppose n ≥ 3. Suppose (d1, . . . , dn) ≺ (a1, . . . , an), where a1 ≥ · · · ≥ an and d1 ≥ · · · ≥ dn. Let k be the largest integer such that ak ≥ d1.

(b.1) If a1 6= an, show that k < n and ak ≥ d1 > ak+1.

Assume a1 ≥ · · · ≥ an ≥ d1 ≥ · · · ≥ dn. Then a1 + ··· + an = d1 + ··· + dn implies that a1 = ··· = an = d1 = ··· = dn, which is a contradiction. ˜ (b.2) Show that d = (d2, . . . , dn) is majorized by a˜ = (a1, . . . , ak−1, a˜k+1, ak+2 . . . , an) where a˜k+1 = ak + ak+1 − d1. Consider the sum of the ` largest entries of the second vector. P` P` P` Case 1. If ` < k, then j=1 aj ≥ j=1 dj ≥ j=1 dj+1, which is the sum of the ` largest entries of the ﬁrst vector. P`+1 Case 2. If ` ≥ k, then the sum of the ` largest entries of a˜ is at least j=1 aj − d1 which is P` ˜ larger than or equal to j=1 dj+1, which is the sum of the ` largest entries of d, by the original assumption. (b.3) (Optional) Show by induction that one can construct a real symmetric matrix with eigenvalues a1, . . . , an and diagonal entries d1, . . . , dn. We prove by induction on n. If n = 2, the result follows from (a). Suppose n > 2. Construct ˜ ∗ the vector d and a˜ in (b). There is unitary (real orthogonal) V ∈ Mn−1 so that V DV˜ has diagonal entries d2, . . . , dn, where D˜ = diag (˜ak+1, a1, . . . , ak−1, ak+2, . . . , an). Now, there is a unitary (real ∗ ak 0 d1 ∗ orthogonal) P ∈ M2 such that P P = . Let Q = (P ⊕ In−2)([1] ⊕ V ). ak+1 0 ∗ a˜k+1 ∗ Then Q diag (ak, ak+1, a1, . . . , ak−1, ak+2, . . . , an)Q has diagonal entries d1, . . . , dn.

Theorem (Lidskii) Let A, B ∈ Mn be Hermitian with eigenvalues a1 ≥ · · · ≥ an and b1 ≥ · · · ≥ bn,

Suppose C = A + B has eigenvalues c1 ≥ · · · ≥ cn. Then for any 1 ≤ i1 < ··· < im ≤ n,

m m m X X X bn−j+1 ≤ (cis − ais ) ≤ bj. s=1 s=1 s=1

Corollary (Weyl) Let A, B ∈ Mn be Hermitian. Then for any j, k ∈ {1, . . . , n} with j + k − 1 ≤ n, we have

λj(A) + λk(B) ≥ λj+k−1(C).

4 Proof. Suppose A has eigenvalues a1 ≥ · · · ≥ an and B has eigenvalues b1 ≥ · · · ≥ bn. We may replace (A, B) by (A − ajI,B − bkI) and assume that aj = 0 = bk. ∗ ∗ Let A = UD1U and B = VD2V be such that U, V ∈ Mn are unitary, D1 = diag (a1, . . . , an) and D2 = diag (b1, . . . , bn). Set

∗ ∗ A1 = Udiag (a1, . . . , aj−1, 0,..., 0)U and B1 = V diag (b1, . . . , bk−1, 0,..., 0)V .

Note that A1 +B1 has rank at most j +k−2, i.e., A1 +B1 has at most j +k−2 nonzero eigenvalues.

Thus, λj+k−1(A1 + B1) = 0 and λj+k−1(A + B) ≤ λ1(A + B − A1 − B1).

Now, set X = A1 + B1, Y = A + B − A1 − B1, and Z = X + Y = A + B. By Lidskii Theorem, we have λj+k−1(Z) − λj+k−1(X) ≤ λ1(Y ).

Next, set X = A − A1, Y = B − B1, and Z = X + Y . By Lidskii Theorem again, λ1(Z) ≤

λ1(X) + λ1(Y ). Thus, λ1(A + B − A1 − B1) ≤ λ1(A − A1) + λ1(B − B1) = λj(A) + λk(B).

Positive semi-deﬁnite matrices

∗ Definition A matrix A ∈ Mn is positive definite (positive semi-definite) if x Ax is positive (nonnegative) for all nonzero x ∈ Cn.

Theorem Let A ∈ Mn. The following are equivalent. (a) A is positive definite (semi-definite). (b) A is normal with positive (nonnegative) eigenvalues. ∗ (c) A = RR , where R ∈ Mn with | det(R)| > 0 and can be chosen to be in triangular form or positive definite (R ∈ Mn with | det(R)| ≥ 0 and can be positive demi-definite).

(d) A is Hermitian with det(Ak) > 0 for k = 1, . . . , n, where Ak is the leading k × k principal t submatrix. (There is a permutation matrix P such that P AP = A˜ ⊕ 0n−m such that det(A˜j) > 0 for j = 1, . . . , m.) Proof. The equivalence of (a) and (b) are shown in class. ∗ To prove (b) ⇒ (c). Let A = UDU where U is unitary, and D = diag (a1, . . . , an). Then for √ √ ∗ ∗ D˜ = diag ( a1,..., an), we have A = RR with R = UDU˜ . Evidently, R is positive deﬁnite. Further, we can write R = LV for some unitary V and lower triangular L so that A = LL∗. To prove (c) ⇒ (a). Note that for any nonzero x ∈ Cn, y = R∗x is nonzero as R is invertible. Thus, x∗Ax = y∗y > 0. Thus, A is positive deﬁnite.

To prove (c) ⇒ (d) if we choose R to be lower triangular in condition (c). Then det(Ak) = ∗ ∗ 2 det(RkRk) = det(Rk) det(Rk) = | det(Rk)| > 0.

To prove (d) ⇒ (c), by Gaussian elimination, we can choose L1 by changing the ﬁrst column of In so that L1A has zero entries in the (i, 1) positions for i = 2, . . . , n. Since A is Hermitian ∗ and the ﬁrst row of L1A is the same as A, we have L1AL1 = [a11] ⊕ A2 where a11 is the (1, 1) ˜ √ ˜ ˜∗ entry of A. Now, change L1 to L1 = L1 diag ( a11, 1,..., 1) . Then L1AL1 = [1] ⊕ A2. Now, use ∗ induction argumet on A2, we see that A2 = L2L2 for some lower triangular matrix with positive ˜−1 ∗ diagonal entries. Set R = L1 ([1] ⊕ L2). Then A = RR .

5 ∗ ∗ Deﬁnition Let A ∈ Mn. Then AA and A A has the same collection of nonnegative eigenvalues. p ∗ The numbers sj(A) = λj(AA ) is the jth singular value of A for j = 1, . . . , n.

Theorem Let A ∈ Mn. (a) (SVD decomposition) There is are unitary matrices such that UAV is a diagonal matrix with diagonal entries s1(A), . . . , sn(A). (b) (Polar decomposition) There are positive semi-deﬁnite matrices P,Q and unitary matrices V,˜ U˜ such that A = P V˜ = UQ˜ .

0 A Theorem Let A ∈ M . Then the matrix has eigenvalues ±s (A),..., ±s (A). n A∗ 0 1 n

Remark We can deduce singular value inequalities using the above (Wielandt) matrix.

Congruence

∗ Theorem Let A ∈ Mn be Hermitian, and S ∈ Mn be invertible. Then S AS and A have the same number of positive and negative eigenvalues. In particular, there is an invertible T ∈ Mn such that ∗ T AT = Ip ⊕ −Iq ⊕ 0s.

∗ Theorem Let A, B ∈ Mn be Hermitian. Then there is an invertible S ∈ Mn such that both S AS and S∗BS are in diagonal form if any one of the following is true. (a) There are real numbers a, b such that aA + bB is positive deﬁnite. (b) The matrix A is invertible and A−1B is similar to a real diagonal matrix.

Symmetric matrices For real symmetric matrices, the theory of Hermitian matrices applies. For complex symmetric matrices, we have the following results.

Theorem Let A ∈ Mn. The following are equivalent. (a) A is symmetric. t (b) There is a unitary U ∈ Mn such that A = UDU such that D has nonnegative entries arranged in descending order. t (c) A = SS for some S ∈ Mn.

Theorem Let A ∈ Mn. Then A is normal and symmetric if and only if there is a real orthogonal matrix A such that QAQt is a diagonal matrix.

Theorem Every matrix A is similar to a symmetric matrix. Consequently, every matrix A is a product of two symmetric matrices.

Theorem Let A ∈ Mn be symmetric. Then A is diagonalizable if and only if there is a complex orthogonal matrix Q such that QtAQ is in diagonal form.