Lecture # 3 Orthogonal Matrices and Matrix Norms We repeat the definition an orthogonal set and orthornormal set. n Definition 1 A set of k vectors fu1; u2;:::; ukg, where each ui 2 R , is said to be an orthogonal with respect to the inner product (·; ·) if (ui; uj) = 0 for i =6 j. The set is said to be orthonormal if it is orthogonal and (ui; ui) = 1 for i = 1; 2; : : : ; k The definition of an orthogonal matrix is related to the definition for vectors, but with a subtle difference. n×k Definition 2 The matrix U = (u1; u2;:::; uk) 2 R whose columns form an orthonormal set is said to be left orthogonal. If k = n, that is, U is square, then U is said to be an orthogonal matrix. Note that the columns of (left) orthogonal matrices are orthonormal, not merely orthogonal. Square complex matrices whose columns form an orthonormal set are called unitary. Example 1 Here are some common 2 × 2 orthogonal matrices ! 1 0 U = 0 1 ! p 1 −1 U = 0:5 1 1 ! 0:8 0:6 U = 0:6 −0:8 ! cos θ sin θ U = − sin θ cos θ Let x 2 Rn then k k2 T Ux 2 = (Ux) (Ux) = xT U T Ux = xT x k k2 = x 2 1 So kUxk2 = kxk2: This property is called orthogonal invariance, it is an important and useful property of the two norm and orthogonal transformations. That is, orthog- onal transformations DO NOT AFFECT the two-norm, there is no compa- rable property for the one-norm or 1-norm. The Cauchy-Schwarz inequality given below is quite useful for all inner products. Lemma 1 (Cauchy-Schwarz inequality) Let (·; ·) be an inner product in Rn. Then for all x; y 2 Rn we have j(x; y)j ≤ (x; x)1=2 (y; y)1=2 : (1) Moreover, equality in (1) holds if and only if x = αy for some α 2 R. In the Euclidean inner product, this is written T jx yj ≤ kxk2kyk2: This inequality leads to the following definition of the angle between two vectors relative to an inner product. Definition 3 The angle θ between the two nonzero vectors x; y 2 Rn with respect to an inner product (·; ·) is given by (x; y) cos θ = : (2) (x; x)1=2 (y; y)1=2 With respect to the Euclidean inner product, this reads xT y cos θ = : kxk2kyk2 The one-norm and the 1-norm share two inequalities similar to the Cauchy{Schwarz inequality. T jx yj ≤ kxk1kyk1; T jx yj ≤ kxk1kyk1; but these do not lead to any reasonable definition of angle between vectors. 2 Example 2 ! ! 1 −1 x = ; y = : 1 3 We have p p T kxk2 = 2; kyk2 = 10; x y = 2 xT y q cos θ = = 1=5: kxk2kyk2 Thus θ = 1:1071R = 63:43◦ is the angle between the two vectors. The three upper bounds on the dot product are p kxk2kyk2 = 20; kxk1kyk1 = 2 · 3 = 6 kxk1kyk1 = 1 · 4 = 4: It is possible to come up with examples where each of these three is the small- est bound (or the largest one). Matrix Norms We will also need norms for matrices. Definition 4 A norm in Rm×n is a function k · k mapping Rm×n into R satisfying the following three axioms 1. kXk ≥ 0; kXk = 0 if and only if X = 0;X 2 Rm×n 2. kαXk = jαjkXk X 2 Rm×n; α 2 R 3. kX + Y k ≤ kXk + kY k X; Y 2 Rm×n. 3 This definition is isomorphic to the definition of a vector norm on Rmn. For example, the Frobenius norm defined by 0 1 1=2 Xm Xn k k @ 2 A X F = xij (3) i=1 j=1 is isomorphic to the two-norm on Rmn. Since X represents a linear operator from Rn to Rm, it is appropriate to m×n define the induced norm k · kα on R by kXykα kXkα = sup : (4) y=06 kykα It is a simple matter to show that kXkα = max kXykα: (5) kykα=1 Note that the maximum is taken over a closed, bounded set, thus we have that ∗ kXkα = kXy kα (6) ∗ ∗ for some y such that ky kα = 1. The above definition leads to the very useful bound kXykα ≤ kXkαkykα (7) where equality occurs for every vector of the form γy∗; γ 2 R. n×n For any induced norm k · k, the identity matrix In for R satisfies kInk = 1: (8) However, for the Frobenius norm p kInkF = n; thus it is not an induced norm for any vector norm. For the one-norm and the 1-norm there are formulas for the correspond- ing matrix norms and for a vector y∗ satisfying (6). The one-norm formula is Xm kXk1 = max jxijj: (9) 1≤j≤n i=1 4 If jmax is the index of a column such that Xm k k j j X 1 = xi;jmax i=1 ∗ then y = ejmax , the corresponding column of the identity matrix. The 1-norm formula is Xn kXk1 = max jxijj: (10) 1≤i≤m j=1 If imax is the index of a row such that Xn k k j j X 1 = ximax;j j=1 ∗ ∗ ∗ T then the vector y = (y1; : : : ; yn) with components ∗ yj = sign(ximax;j) T satisfies (6). Note that kXk1 = kX k1. The matrix two-norm does not have a formula like (9) or (10) and all other formulations are really equivalent to (4). Moreover, computing the vector y∗ in (6) is a nontrivial task that we will discuss later. The induced norms have a convenient property that is important in un- derstanding matrix computations. For X 2 Rm×n and Y 2 Rn×s consider kXY kα. We have that kXY kα = max kXY zkα ≤ max kXkαkY zkα kzkα=1 kzkα=1 = kXkα max kY zkα = kXkαkY kα: kzkα=1 Thus kXY kα ≤ kXkαkY kα: (11) A norm k · kα (or really family of norms) that satisfies the property (11) is said to be consistent. Since they are induced norms the two-norm, one-norm, and the 1-norm are all consistent. The Frobenius norm also satisfies (11). An example of a matrix norm that is not consistent is given below. 5 m×n Example 3 Consider the norm k · kβ on R given by kXkβ = max jxijj: (i;j) This is simply the 1-norm applied to X written out as vector in Rmn. For m = n = 2, consider ! 1 1 X = Y = : 1 1 Note that ! 2 2 XY = 2 2 and thus kXY kβ = 2 > kXkβkY kβ = 1. Clearly, k · kβ is not consistent. Henceforth, we use only consistent norms. Now we give a numerical example with our four most used norms. Example 4 Consider 0 1 3 −2 1 B C X = @ 10 0 −16 A : −3 25 1 It is easily verified that 0 1 0 B C k k ∗ @ A X 1 = 27; y1 = e2 = 1 ; 0 0 1 −1 ∗ B C kXk1 = 29: y1 = @ 1 A ; 1 kXkF = 31:70; kXk2 = 25:46: The \magic vector" in the two-norm is (to the digits displayed) 0 1 −0:18943 B C ∗ @ A y2 = 0:97256 0:13508 This will not always be true, but notice that its sign pattern is the same ∗ as y1 and that its largest component corresponds to the non-zero component ∗ of y1. 6.