Introduction to Linear Algebra Tyrone L. Vincent Engineering Division, Colorado School of Mines, Golden, CO E-mail address: [email protected] URL: http://egweb.mines.edu/~tvincent Contents Chapter 1. Revew of Vectors and Matricies 1 1. Useful Notation 1 2. Vectors and Matricies 2 3. Basic operations 3 4. Useful Properties of the Basic Operations 6 Chapter 2. Vector Spaces 9 1. Vector Space De…nition 9 2. Linear Independence and Basis 10 3. Change of basis 12 4. Norms, Dot Products, Orthonormal Basis 14 5. QR Decomposition 16 Chapter 3. Projection Theorem 21 Chapter 4. Matrices and Linear Mappings 23 1. Solutions to Systems of Linear Equations 24 Chapter 5. Square Matrices, Eigenvalues and Eigenvectors 25 1. Matrix Exponential 26 2. Other Matrix Functions 26 Appendix A. Appendix A 29 iii CHAPTER 1 Revew of Vectors and Matricies 1. Useful Notation 1.1. Common Abbrivations. In this course, as in most branches of math- ematics, we will often utilize sets of mathematical objects. For example, there is the set of natural numbers, which begins 1; 2; 3; : This set is often denoted N, so that 2 is a member of N but is not. To specify that an object is a member of a set, we use the notation for "is a member of". For example 2 N: Some of the sets we will use are 2 2 R real numbers C complex numbers Rn n dimensional vectors of real numbers m n R m n dimensional real matrices For these common sets, particular notation will be used to identfy members, namely lower case for a scalar or vector, and upper case for a matrix. The following table also includes some common operations x vector or scalar x; y inner product between vectors x and y Ah i matrix AT transpose of A 1 A inverse of A det(A), A determinant of A j j To specify a set, we can also use a bracket notation. For example, to specify E as the set of all positive even numbers, we can say either E = 2; 4; 6; 8; f g when the pattern is clear, or use a : symbol, which means "such that": E = x N : mod(x; 2) = 0 : f 2 g This can be read "The set of natural numbers x, such that x is divisible evenly by 2". When talking about sets, we will often want to say when a property holds for every member of the set, or for at least one. In this case, the symbol ; meaning 8 "for all" and ; meaning "there exists" are useful. For example, suppose I is the set numbers consisting9 of the IQs for people in this class. Then x I x > 110 8 2 1 2 1.REVEWOFVECTORSANDMATRICIES means that all students in this class have IQ greater than 110 while x I : x > 110 9 2 means that at least one student in the class has IQ greater than 110. We will also be concerned with functions. Given a set X and a set Y a function f from X to Y maps an element from X to and element of Y and is denoted f : X Y: ! The set X is called the domain, and f(x) is assumed to be de…ned for every x X: The range, or image of f is the set of y for which f(x) = y for some x : 2 Range(f) = y Y : x X such that y = f(x) : f 2 9 2 g If Range(f) = Y; then f is called "onto". If there is only one x X such that y = f(x); then f is called "one to one". 2 2. Vectors and Matricies You are probably already familiar with vectors and matrices from previous courses in mathmatics or physics. We will …nd matrices and vectors very useful when representing dynamic systems mathematically, however, we will need to be able to manipulate and understand these objects at a fairly deep level. Some texts use bold face for vectors and matrices, but ours does not, and I will not use that convention here, or during class. I will however, use lower case letters for vectors and upper case letters for matrices. A vector of n tuple real (or sometimes complex) numbers is represented as: x1 x2 x = 2 . 3 . 6 7 6 x 7 6 n 7 4 5 n So that x is a vector, and xi are each scalars. We will use the notation xi R to n 2 show that x is a length p vector of real numbers (or xi C if the elements of xi are complex.) Sometimes we will want to index vectors as2 well, which can sometimes be confusing: Is xi the vector xi or the ith element of the vector x? To make the di¤erence clear, we will reserve the notation [x]i to indicate the ith element of x: As an example, consider the following illustration of addition and scalar multiplication for vectors: [x1]1 + [x2]1 [x1]1 [x1]2 + [x2]2 [x1]2 x1 + x2 = 2 . 3 x1 = 2 . 3 . 6 7 6 7 6 [x ] + [x ] 7 6 [x ] 7 6 1 n 2 n 7 6 1 n 7 A matrix is an m n4array of scalars:5 4 5 a11 a12 a1n a21 a22 a2n A = 2 . . 3 . .. 6 7 6 am1 am2 amn 7 6 7 m 4n 5 We use the notation A R indicate that A is a m n matrix. Addition and scalar multiplication are2 de…ned the same way as for vectors. 3.BASICOPERATIONS 3 3. Basic operations You should already be faimilar with most of the basic operations on vectors and matricies listed in this section. m n T n m 3.1. Transpose. Given a matrix A R ; the transpose A R is found by ‡ipping all terms along the diagonal.2 That is, if 2 a11 a12 a1n a21 a22 a2n A = 2 . . 3 . .. 6 7 6 am1 am2 amn 7 6 7 then 4 5 a11 a21 an1 a12 a22 an2 T A = 2 . . 3 . .. 6 7 6 a1m a2m anm 7 6 7 Note that if the matrix is not4 square (m = n); then the5 “shape” of the matrix 6 changes. We can also use transpose on a vector x Rn by considering it to be an n by 1 matrix. In this case, xT is the 1 by n matrix2: T x = [x]1 [x]2 [x]n 3.2. Inner (dot) product. In three dimensional space, we are familiar with vectors as indicating direction. The inner product is an operation that allows us to tell if two vectors are pointing in a similar direction. We will use the notation x; y for inner product between x and y: In other courses, you may have seen this calledh i the dot product with notation x y: The notation used here is more common in signal processing and control systems. The inner product of x; y Rn is de…ned to be the sum of product of the elements 2 n x; y = [x]i[y]i h i i=1 X = xT y Recall that if x and y are vectors, the angle between them can be found using the formula x; y cos = h i x; x y; y h i h i Note that the inner product satis…es thep following rules (inherited from transpose) x + y; z = x; z + y; z h i h i h i y; z = y; z h i h i 3.3. Matrix-vector multiplication. Suppose we have an m n matrix A a11 a12 a1n a21 a22 a2n A = 2 . . 3 . .. 6 7 6 am1 am2 amn 7 6 7 4 5 4 1.REVEWOFVECTORSANDMATRICIES and a length n vector x1: Note that the number of columns of A are the same as the length of x1: Multiplication of A and x1 is de…ned as follows: x2 = Ax1 (3.1) where [x2]1 = a11[x1]1 + a12[x1]2 + + a1n[x1]n (3.2a) [x2]2 = a21[x1]1 + a22[x1]2 + + a2n[x1]n (3.2b) . (3.2c) [x2]m = am1[x1]1 + am2[x1]2 + + amn[x1]n (3.2d) Note that the result x2 is a length m vector (the number of rows of A). The notation (3.1) is a compact representation of the system of linear algebraic equations (3.2). Note that A de…nes a mapping from Rn to Rm: Thus, we can write A : Rn Rm: This mapping is linear. ! We can also consider a matrix to be a group of vectors. For example, if we group the vectors x1; x2; ; xn into a matrix M = x1 x2 xp and de…ne the vector 1 2 a = 2 . 3 . 6 7 6 7 6 p 7 Then all linear combinations of x1; x2; 4 ; xp5are given by y = Ma = 1x1 + 2x2 + + pxp 3.4. Matrix-Matrix multiplication. If matrix A : Rn Rm; and matrix B : Rm Rp; we can …nd mapping C: Rn Rp which is the! composition of A and B ! ! C = BA [c]ij = [b]ik [a]kj k X That is, the i; j element of C is the dot product of the ith row of B with the jth column of A: The dimension of C is p n: This can also be though of as B mapping a column of A at a time: That is, the …rst column of C; [c] 1 is B[a] 1;B times the …rst column of A: Clearly, two matricies can be multiplied only if they have compatible dimensions.