Lecture 5a Linear Algebra Review W. Gambill Department of Computer Science University of Illinois at Urbana-Champaign February 2, 2010 W. Gambill (UIUC) CS 357 February 2, 2010 1 / 74 Vector Space Example The set of n-tuples in Rn form a vector space. 2 3 v1 6 v2 7 = 6 7 v 6 . 7 4 . 5 vn By convention we will write vectors in column form. The transpose operator T converts column vectors to row vectors and vice versa. 2 3T v1 6 v2 7 6 7 = ... 6 . 7 v1 v2 vn 4 . 5 vn W. Gambill (UIUC) CS 357 February 2, 2010 2 / 74 Vector Space Example 2 The set of continuous functions defined on a closed interval e.g. f 2 C([a, b]), f :[a, b] ! R form a vector space over the reals R. If f1, f2 2 C([a, b]) then f1 + f2 2 C([a, b]) If r 2 R then r ∗ f1 2 C([a, b]) W. Gambill (UIUC) CS 357 February 2, 2010 3 / 74 Vector Operations Addition and Subtraction Multiplication by a scalar Transpose Linear Combinations of Vectors Inner Product Outer Product Vector Norms W. Gambill (UIUC) CS 357 February 2, 2010 4 / 74 Vector Addition and Subtraction Addition and subtraction are element-by-element operations c = a + b () ci = ai + bi i = 1,..., n d = a - b () di = ai - bi i = 1,..., n 213 233 a = 425 b = 425 3 1 243 2 -2 3 a + b = 445 a - b = 4 0 5 4 2 W. Gambill (UIUC) CS 357 February 2, 2010 5 / 74 Multiplication by a Scalar Multiplication by a scalar involves multiplying each element in the vector by the scalar: b = σa () bi = σai i = 1,..., n 243 223 a a = 465 b = = 435 8 2 4 W. Gambill (UIUC) CS 357 February 2, 2010 6 / 74 Vector Transpose The transpose of a row vector is a column vector: 213 T u = 1, 2, 3 then u = 425 3 Likewise if v is the column vector 243 T v = 455 then v = 4, 5, 6 6 W. Gambill (UIUC) CS 357 February 2, 2010 7 / 74 Linear Combinations Combine scalar multiplication with addition 2 3 2 3 2 3 2 3 u1 v1 αu1 + βv1 w1 6u2 7 6v2 7 6 αu2 + βv2 7 6w2 7 α 6 7 + β 6 7 = 6 7 = 6 7 6 . 7 6 . 7 6 . 7 6 . 7 4 . 5 4 . 5 4 . 5 4 . 5 um vm αum + βvm wm 2 -2 3 213 r = 4 1 5 s = 405 3 3 2 -4 3 233 2 -1 3 t = 2r + 3s = 4 2 5 + 405 = 4 2 5 6 9 15 W. Gambill (UIUC) CS 357 February 2, 2010 8 / 74 Linear Combinations Any one vector can be created from an infinite combination of other “suitable” vectors. 4 1 0 w = = 4 + 2 2 0 1 1 1 w = 6 - 2 0 -1 2 -1 w = - 2 4 1 4 1 0 w = 2 - 4 - 2 2 0 1 W. Gambill (UIUC) CS 357 February 2, 2010 9 / 74 Linear Combinations 4 [2,4] [1,-1] Graphical interpretation: 3 Vector tails can be moved to convenient 2 [4,2] locations [-1,1] Magnitude and 1 direction of vectors is [0,1] [1,1] preserved 0 [1,0] 0 1 2 3 4 5 6 W. Gambill (UIUC) CS 357 February 2, 2010 10 / 74 Vector Inner Product In physics, analytical geometry, and engineering, the dot product has a geometric interpretation n σ = x · y () σ = xiyi = Xi 1 x · y = kxk2 kyk2 cos θ W. Gambill (UIUC) CS 357 February 2, 2010 11 / 74 Vector Inner Product The inner product of x and y requires that x be a row vector y be a column vector 2 3 y1 6y27 x1 x2 x3 x4 6 7 = x1y1 + x2y2 + x3y3 + x4y4 4y35 y4 W. Gambill (UIUC) CS 357 February 2, 2010 12 / 74 Vector Inner Product For two n-element column vectors, u and v, the inner product is n T σ = u v () σ = uivi = Xi 1 The inner product is commutative so that (for two column vectors) uTv = vTu W. Gambill (UIUC) CS 357 February 2, 2010 13 / 74 Computing the Inner Product in Matlab The * operator performs the inner product if two vectors are compatible. 1 >> u = (0:3)';%u andv are 2 >> v = (3:-1:0)';% column vectors 3 >> s = u*v 4 ??? Error using ==> * 5 Inner matrix dimensions must agree. 6 7 >> s = u'*v 8 s = 9 4 10 11 >> t = v'*u 12 t = 13 4 14 15 >> dot(u,v) 16 ans= 17 4 W. Gambill (UIUC) CS 357 February 2, 2010 14 / 74 Vector Outer Product The inner product results in a scalar. Example The outer product creates a rank- one matrix: Outer product of two 4-element column vectors T A = uv () ai,j = uivj 2 3 u1 T 6u27 uv = 6 7 v1 v2 v3 v4 4u35 u4 2 3 u1v1 u1v2 u1v3 u1v4 6u2v1 u2v2 u2v3 u2v47 = 6 7 4u3v1 u3v2 u3v3 u3v45 u4v1 u4v2 u4v3 u4v4 W. Gambill (UIUC) CS 357 February 2, 2010 15 / 74 Computing the Outer Product in Matlab The * operator performs the outer product if two vectors are compatible. 1 u = (0:4) '; 2 v = (4:-1:0)'; 3 A = u*v' 4 A = 5 0 0 0 0 0 6 4 3 2 1 0 7 8 6 4 2 0 8 12 9 6 3 0 9 16 12 8 4 0 W. Gambill (UIUC) CS 357 February 2, 2010 16 / 74 Vector Norms Compare magnitude of scalars with the absolute value α > β Compare magnitude of vectors with norms kxk > kyk There are several ways to compute jjxjj. In other words the size of two vectors can be compared with different norms. W. Gambill (UIUC) CS 357 February 2, 2010 17 / 74 Vector Norms Consider two element vectors, which lie in a plane b = (2,4) a = (4,2) c = (2,1) a = (4,2) Use geometric lengths to represent the magnitudes of the vectors p p p p 2 2 p 2 2 p 2 2 `a = 4 + 2 = 20, `b = 2 + 4 = 20, `c = 2 + 1 = 5 We conclude that `a = `b and `a > `c or kak = kbk and kak > kck W. Gambill (UIUC) CS 357 February 2, 2010 18 / 74 The L2 Norm The notion of a geometric length for 2D or 3D vectors can be extended vectors with arbitrary numbers of elements. The result is called the Euclidian or L2 norm: n !1=2 2 2 21=2 2 kxk2 = x1 + x2 + ... + xn = xi = Xi 1 The L2 norm can also be expressed in terms of the inner product p p T kxk2 = x · x = x x W. Gambill (UIUC) CS 357 February 2, 2010 19 / 74 p-Norms For any positive integer p p p p1=p kxkp = jx1j + jx2j + ... + jxnj The L1 norm is sum of absolute values n kxk1 = jx1j + jx2j + ... + jxnj = jxij = Xi 1 The L norm or max norm is 1 kxk = max (jx1j, jx2j,..., jxnj) = max (jxij) i 1 Although p can be any positive number, p = 1, 2, are most commonly used. 1 W. Gambill (UIUC) CS 357 February 2, 2010 20 / 74 Application of Norms Are two vectors (nearly) equal? Floating point comparison of two scalars with absolute value: α - β < δ α where δ is a small tolerance. Comparison of two vectors with norms: ky - zk < δ kzk W. Gambill (UIUC) CS 357 February 2, 2010 21 / 74 Application of Norms Notice that ky - zk < δ kzk is not equivalent to kyk - kzk < δ. kzk This comparison is important in convergence tests for sequences of vectors. W. Gambill (UIUC) CS 357 February 2, 2010 22 / 74 Application of Norms Creating a Unit Vector T Given u = [u1, u2,..., um] , the unit vector in the direction of u is u uˆ = kuk2 Proof: u 1 kuˆk2 = = kuk2 = 1 kuk2 2 kuk2 The following are not unit vectors u u kuk1 kuk 1 W. Gambill (UIUC) CS 357 February 2, 2010 23 / 74 Orthogonal Vectors From geometric interpretation of the inner product u · v = kuk2 kvk2 cos θ u · v uTv cos θ = = kuk2 kvk2 kuk2 kvk2 Two vectors are orthogonal when θ = π/2 or u · v = 0. In other words uTv = 0 if and only if u and v are orthogonal. W. Gambill (UIUC) CS 357 February 2, 2010 24 / 74 Orthonormal Vectors Orthonormal vectors are unit vectors that are orthogonal. A unit vector has an L2 norm of one. The unit vector in the direction of u is u uˆ = kuk2 Since p kuk2 = u · u it follows that u · u = 1 if u is a unit vector. W. Gambill (UIUC) CS 357 February 2, 2010 25 / 74 Matrices Columns and Rows of a Matrix are Vectors Addition and Subtraction Multiplication by a scalar Transpose Linear Combinations of Vectors Matrix–Vector Product Matrix–Matrix Product W. Gambill (UIUC) CS 357 February 2, 2010 26 / 74 Notation The matrix A with m rows and n columns looks like: 2 3 a11 a12 ··· a1n 6a21 a22 a2n 7 = 6 7 A 6 . 7 4 . 5 am1 ··· amn aij = element in row i, and column j In Matlab we can define a matrix with 1 >> A = [ ... ; ... ; ... ] where semicolons separate lists of row elements. The a2,3 element of the Matlab matrix A is A(2,3). W. Gambill (UIUC) CS 357 February 2, 2010 27 / 74 Matrices Consist of Row and Column Vectors As a collection of column vectors As a collection of row vectors 2 3 0 2 a 3 (1) 6 7 6 7 6 7 6 7 A = 6a(1) a(2) ··· a(n)7 6 0 7 6 7 6 a(2) 7 4 5 6 7 6 7 A = 6 7 6 .