Matrix Norms Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo September 28, 2009 Matrix Norms We consider matrix norms on (Cm;n; C). All results holds for (Rm;n; R). Definition (Matrix Norms) A function k·k: Cm;n ! C is called a matrix norm on Cm;n if for all A; B 2 Cm;n and all α 2 C 1. kAk ≥ 0 with equality if and only if A = 0. (positivity) 2. kαAk = jαj kAk. (homogeneity) 3. kA + Bk ≤ kAk + kBk. (subadditivity) A matrix norm is simply a vector norm on the finite dimensional vector spaces (Cm;n; C) of m × n matrices. Equivalent norms Adapting some general results on vector norms to matrix norms give Theorem x 1. All matrix norms are equivalent. Thus, if k·k and k·k0 are two matrix norms on Cm;n then there are positive constants µ and M such that µkAk ≤ kAk0 ≤ MkAk holds for all A 2 Cm;n. 2. A matrix norm is a continuous function k·k: Cm;n ! R. Examples mn I From any vector norm k kV on C we can define a m;n matrix norm on C by kAk := kvec(A)kV , where vec(A) 2 Cmn is the vector obtained by stacking the columns of A on top of each other. I m n X X kAkS := jaij j; p = 1, Sum norm; i=1 j=1 m n X X 21=2 kAkF := jaij j ; p = 2, Frobenius norm; i=1 j=1 kAkM := maxjaij j; p = 1, Max norm: i;j (1) The Frobenius Matrix Norm 1. H 2 Pn Pm 2 Pm Pn 2 2 I kA kF = j=1 i=1jaij j = i=1 j=1jaij j = kAkF . The Frobenius Matrix Norm 2. Pm Pn 2 Pm 2 I i=1 j=1jaij j = i=1kai·k2 Pm Pn 2 Pn Pm 2 Pn 2 I i=1 j=1jaij j = j=1 i=1jaij j = j=1ka·j k2. Unitary Invariance. m;n m;m n;n I If A 2 C and U 2 C , V 2 C are unitary 2 2: Pn 2 Pn 2 2: 2 I kUAkF = j=1kUa·j k2 = j=1ka·j k2 = kAkF : 1: H H H 1: I kAVkF = kV A kF = kA kF = kAkF . Submultiplicativity I Suppose A; B are rectangular matrices so that the product AB is defined. 2 Pn Pk T 2 I kABkF = i=1 j=1 ai· b·j ≤ Pn Pk 2 2 2 2 i=1 j=1kai·k2kb·j k2 = kAkF kBkF : Subordinance n I kAxk2 ≤ kAkF kxk2, for all x 2 C . I Since kvkF = kvk2 for a vector this follows from submultiplicativity. Explicit Expression m;n I Let A 2 C have singular values σ1; : : : ; σn and SVD A = UΣVH . Then 3: H p 2 2 I kAkF = kU AVkF = kΣkF = σ1 + ··· + σn. Consistency n;n I A matrix norm is called consistent on C if 4. kABk ≤ kAk kBk (submultiplicativity) holds for all A; B 2 Cn;n. m;n I A matrix norm is consistent if it is defined on C for all m; n 2 N, and 4. holds for all matrices A; B for which the product AB is defined. I The Frobenius norm is consistent. I The Sum norm is consistent. I The Max norm is not consistent. p m;n I The norm kAk := mnkAkM ; A 2 C is consistent. Subordinate Matrix Norm Definition I Suppose m; n 2 N are given, m n I Let k kα on C and k kβ on C be vector norms, and let k k be a matrix norm on Cm;n. I We say that the matrix norm k k is subordinate to the vector norms k kα and k kβ if kAxkα ≤ kAk kxkβ for all A 2 Cm;n and all x 2 Cn. I If k kα = k kβ then we say that k k is subordinate to k kα. I The Frobenius norm is subordinate to the Euclidian vector norm. I The Sum norm is subordinate to the l1-norm. I kAxk1 ≤ kAkM kxk1. Operator Norm Definition Suppose m; n 2 N are given and let k·kα be a vector norm on n m m;n C and k·kβ a vector norm on C . For A 2 C we define kAxkβ kAk := kAkα,β := max : (2) x6=0 kxkα We call this the (α; β) operator norm, the (α; β)-norm, or simply the α-norm if α = β. Observations kAxkα I kAkα,β = maxx2=ker(A) = maxkxk =1kAxkα: kxkβ β I kAxkα ≤ kAkkxkβ. ∗ ∗ n ∗ I kAkα,β = kAx kα for some x 2 C with kx kβ = 1. mn; I The operator norm is a matrix norm on C . I The Sum norm and Frobenius norm are not an α operator norm for any α. Operator norm Properties m;n I The operator norm is a matrix norm on C . I The operator norm is consistent if the vector norm k kα is defined for all m 2 N and k kβ = k kα. Proof In 2. and 3. below we take the max over the unit sphere Sβ. 1. Nonnegativity is obvious. If kAk = 0 then kAykβ = 0 for n each y 2 C . In particular, each column Aej in A is zero. Hence A = 0. 2. kcAk = maxxkcAxkα = maxxjcj kAxkα = jcj kAk. 3. kA + Bk = maxxk(A + B)xkα ≤ maxxkAxkα + maxxkBxkα = kAk + kBk: 4. kABk = max kABxkα = max kABxkα kBxkα Bx6=0 kxkα Bx6=0 kBxkα kxkα ≤ max kAykα max kBxkα = kAk kBk: y6=0 kykα x6=0 kxkα The p matrix norm I The operator norms k·kp defined from the p-vector norms are of special interest. I Recall Pn p1=p kxkp := j=1jxj j ; p ≥ 1; kxk1 := max1≤j≤njxj j: I Used quite frequently for p = 1; 2; 1. I We define for any 1 ≤ p ≤ 1 kAxkp kAkp := max = max kAykp: (3) x6=0 kxkp kykp=1 I The p-norms are consistent matrix norms which are subordinate to the p-vector norm. Explicit expressions Theorem For A 2 Cm;n we have m X kAk1 := max jak;j j; (max column sum) 1≤j≤n k=1 kAk2 := σ1; (largest singular value of A) n X kAk1 := max jak;j j; (max row sum): 1≤k≤m j=1 (4) The expression kAk2 is called the two-norm or the spectral norm of A. The explicit expression follows from the minmax theorem for singular values. Examples 1 14 4 16 For A := 15 [ 2 22 13 ] we find 29 I kAk1 = 15 . I kAk2 = 2. 37 I kAk1 = . p15 I kAkF = 5. 1 2 I A := [ 3 4 ] I kAk1 = 6 I kAk2 = 5:465 I kAk1 = 7. I kAkF = 5:4772 The 2 norm Theorem n;n Suppose A 2 C has singular values σ1 ≥ σ2 ≥ · · · ≥ σn and eigenvalues jλ1j ≥ jλ2j ≥ · · · ≥ jλnj. Then −1 1 kAk2 = σ1 and kA k2 = ; (5) σn −1 1 kAk2 = λ1 and kA k2 = ; if A is symmetric positive definite; λn (6) −1 1 kAk2 = jλ1j and kA k2 = ; if A is normal: (7) jλnj For the norms of A−1 we assume of course that A is nonsingular. Unitary Transformations Definition A matrix norm k k on Cm;n is called unitary invariant if kUAVk = kAk for any A 2 Cm;n and any unitary matrices U 2 Cm;m and V 2 Cn;n. If U and V are unitary then U(A + E)V = UAV + F, where kFk = kEk. Theorem The Frobenius norm and the spectral norm are unitary H H invariant. Moreover kA kF = kAkF and kA k2 = kAk2. Proof 2 norm I kUAk2 = maxkxk2=1kUAxk2 = maxkxk2=1kAxk2 = kAk2: H I kA k2 = kAk2 (same singular values). H H H H I kAVk2 = k(AV) k2 = kV A k2 = kA k2 = kAk2: Perturbation of linear systems I x1 + x2 = 20 −16 −15 x1 + (1 − 10 )x2 = 20 − 10 I The exact solution is x1 = x2 = 10. I Suppose we replace the second equation by −16 −15 x1 + (1 + 10 )x2 = 20 − 10 ; I the exact solution changes to x1 = 30, x2 = −10. −16 I A small change in one of the coefficients, from 1 − 10 to 1 + 10−16, changed the exact solution by a large amount. Ill Conditioning I A mathematical problem in which the solution is very sensitive to changes in the data is called ill-conditioned or sometimes ill-posed. I Such problems are difficult to solve on a computer. I If at all possible, the mathematical model should be changed to obtain a more well-conditioned or properly-posed problem. Perturbations I We consider what effect a small change (perturbation) in the data A,b has on the solution x of a linear system Ax = b. I Suppose y solves (A + E)y = b+e where E is a (small) n × n matrix and e a (small) vector. I How large can y−x be? I To measure this we use vector and matrix norms. Conditions on the norms n I k·k will denote a vector norm on C and also a submultiplicative matrix norm on Cn;n which in addition is subordinate to the vector norm. n;n n I Thus for any A; B 2 C and any x 2 C we have kABk ≤ kAk kBk and kAxk ≤ kAk kxk: I This is satisfied if the matrix norm is the operator norm corresponding to the given vector norm or the Frobenius norm.