Topics in Applied Linear Algebra - Part II April 23, 2013 Some Preliminary Remarks The purpose of these notes is to provide a guide through the material for the second part of the graduate module HM802 Topics in Applied Linear Algebra. The material presented is by and large classical and the exposition draws heavily on the texts [1, 2, 3]. The proofs of a number of results are included in full; where proofs are omitted, the student is encouraged to either provide the details themselves or to consult the above references to find a proof. Typically, if a proof is requested as an exercise, the student should at least attempt to prove the result themselves before consulting the texts. In addition to the exercises provided here, sample problems will be circulated and these should be worked through during the period between the end of the classes and the examination. Finally, I am confident that the notes contain omissions and typos, and I will be grateful if you communicate these to me as you find them! 1 Chapter 1 Norms, Spectral Radii and Eigenvalues 1.1 Definitions, Examples and Elementary Results It is very important for many applications to be able to describe the size of vectors and matrices. For instance, in numerical analysis, we often need to be able to quantify the size of errors that arise in the course of running the algorithm. Similarly to study the rate at which iterative methods converge, some means of measuring the distance to the final, steady state vector is required. The same issue is of central important in stability theory and applications in control. Norms can be used to measure the length or size of vectors or matrices. In general, there are a variety of norms that can be defined on a vector space and the general definition given below tries to capture the essential features of our notions of length and size. Definition 1.1.1 A norm k · k on a vector space V is a function from V to R satisfying the following properties: (i) kxk ≥ 0 for all x 2 V ; (ii) kxk = 0 if and only if x = 0; (iii) kλxk = jλjkxk for all x 2 V and λ 2 C; (iv) kx + yk ≤ kxk + kyk for all x; y in V . In this course, we shall only be concerned with norms on finite dimensional vector spaces. In fact, the norms we study will be defined on either the n n n×n n×n spaces R , C or the matrix spaces R , C . Before discussing some well-known examples of norms, we note the following simple fact. 2 Lemma 1.1.1 Let k · k be a norm on a vector space V . Then for any x; y 2 V , jkxk − kykj ≤ kx − yk: Proof: As x = y + (x − y), the triangle inequality implies that kxk ≤ kyk + kx − yk from which we see that kxk − kyk ≤ kx − yk: (1.1) A similar calculation using y = x + (y − x) yields kyk − kxk ≤ kx − yk: (1.2) Combining (1.1), (1.2) yields the result. n n Example 1.1.1 The most familiar norm on C (or R ) is the Euclidean or l2 norm given by v u n uX 2 kxk2 = t jxij : (1.3) i=1 Two other well-known norms are the infinity norm kxk1 = max jxij; (1.4) 1≤i≤n and the l1 norm n X kxk1 = jxij: (1.5) i=1 The three norms introduced above are all special cases of a general family of norms given by n !1=p X p kxkp = jxij : (1.6) i=1 Proving that the l2, l1 and l1 norms satisfy the axioms of a norm is relatively straightforward with the triangle inequality the only non-trivial exercise in each case. For the general lp norm, proving the triangle inequality is a little more involved and the standard method of proof relies on using H¨older's inequality or properties of convex functions. Exercise: Verify that the l1 and l1 norms all satisfy the axioms of a norm. Inner Products 3 The l2 norm has a special property; it arises naturally from the usual `dot- product' n ∗ X hz; wi = w z = w¯izi: i=1 This is an example of an inner product, the general definition of which is given below. Definition 1.1.2 An inner product h·; ·i on a vector space V is a mapping from V × V to C with the properties: (i) hz + v; wi = hz; wi + hv; wi; (ii) hcz; wi = chz; wi; (iii) hw; zi = hz; wi; (iv) hz; zi ≥ 0 and hz; zi = 0 if and only if z = 0. The following fundamental, and well-known, fact about inner products is known as the Cauchy-Schwartz inequality. Proposition 1.1.1 Let h·; ·i be an inner product on V . Then for any x; y in V , jhx; yij2 ≤ hx; xihy; yi: (1.7) Moreover, the above inequality is strict unless y and x are linearly dependent. Proof: If y is zero, the result is trivial so we assume that y 6= 0. Then for any real number t hx + ty; x + tyi ≥ 0: Expanding this expression shows that t2hy; yi + 2tRe(hx; yi) + hx; xi ≥ 0 for all t 2 R. As the above quadratic expression is non-negative for all real t, we must have 4(Re(hx; yi)2 − 4hy; yihx; xi ≤ 0: To see that (1.7) holds, note that the above inequality must also be true if we replace y with hx; yiy. This implies that 4(Re(hx; yihx; yi))2 ≤ 4jhx; yij2hy; yihx; xi: This immediately yields (1.7). Finally, note that the inequality is strict unless for some t, hx + ty; x + tyi = 0: 4 This implies that x + ty = 0 and hence x; y are linearly dependent. There is a norm on V naturally associated with any inner product h·; ·i, given by kxk = phx; xi. Exercise: Prove that kxk = phx; xi satisfies the axioms of a norm on V . Exercise: Show that for any two norms k·ka, k·kb, kxk = maxfkxka; kxkbg is also a norm. Exercise: Show that for any norm k · k, and non-singular linear operator or matrix T , kxkT = kT xk is a norm. 3 Exercise: Show that the mapping f on R defined by f(x1; x2; x3) = p 2 2 2 (2x1) + (x1 − 3x2) + (3x1 + x2 − 2x3) is a norm. Continuity We have introduced a norm k·k as a function from V to R. For convenience, we shall write F to denote either the base field R or C. Also recall that we can naturally map any finite n-dimensional vector space isomorphically to n F by fixing a basis. n The next result notes that any norm on F is continuous with respect to the simple l1 norm above. This elementary fact has important consequences and significantly simplifies the study of convergence for iterative processes and dynamical systems in finite dimensions. n Proposition 1.1.2 Let k · k be any norm on F . Then k · k is Lipschitz continuous with respect to the infinity norm k · k1. n Proof: Let x; y 2 F be given. It follows from Lemma 1.1.1 that jkxk − kykj ≤ kx − yk: Next, apply the triangle inequality and the homogeneity property of norms to conclude that n X kx − yk ≤ jxi − yijkeik i=1 n where ei is the ith standard basis vector in F . If we combine the previous two inequalities, we see immediately that jkxk − kykj ≤ Kkx − yk1 Pn where K = i=1 keik: This completes the proof. We can now use the previous proposition and the fact that the unit ball in the 1-norm is a compact set to prove the following key fact. 5 n Theorem 1.1.1 Let k · ka, k · kb be two norms on F . There exist positive constants m and M such that mkxka ≤ kxkb ≤ Mkxka (1.8) n for all x 2 F . Proof: If x = 0, then the result is trivial. Let B denote the unit ball in the 1-norm, n B := fx 2 F j kxk1 = 1g: Then B is a compact set and moreover, we know that kxka > 0, kxkb > 0 for all x 2 B. Consider the function kxk φ(x) = b kxka defined for x 2 B. From the above observations, φ is well-defined. Moreover, Proposition 1.1.2 implies that φ as the quotient of 2 continuous functions is continuous. Hence φ is lower and upper bounded on B and attains these bounds. Formally, there exist m > 0, M > 0 with m ≤ φ(x) ≤ M for all x 2 B. Rewriting this, we see that mkxka ≤ kxkb ≤ Mkxka for all x 2 B. The result follows from the observation that for any non-zero x, x 2 B. kxk1 n Theorem 1.1.1 shows that any two norms on F are equivalent. This has n important implications for the study of convergence in F . As usual, a n n sequence xn in F converges to a point x 2 F with respect to a norm k · k if kxn − xk ! 0 as n ! 1. It follows readily from Theorem 1.1.1 that the choice of norm is not important in determining the convergence of a sequence. Formally, given two norms k · ka, k · kb, xn converges to x with respect to k · ka if and only if xn converges to x with respect to k · kb. Exercise: Find constants m and M such that the inequalities mkxk2 ≤ kxk1 ≤ Mkxk2 hold and are tight.