ICME Linear Algebra Refresher Course Lecture 1: Preliminaries Ron Estrin September 20, 2016 1/47 Ron Estrin Lecture 1: Preliminaries Introduction This course: A short refresher on linear algebra, meant to prepare you for CME 302, CME 200, or other courses involving linear algebra. Prerequisites: Some level of exposure to linear algebra in your undergrad career. Hopefully most of what you'll see is review, but if we're ever going too fast (or slow), ask a question! When: Tues/Wed/Thurs, 10:30am - 11:45am. Slides and material accessible at: http://stanford.edu/~tym1/refresher/index.html. Much of the material is shamelessly re-used from offerings of previous years (in particular, Victor Minden's slides 2014 slides). 2/47 Ron Estrin Lecture 1: Preliminaries Useful Resources Matrix Computations 3ed by Golub and Van Loan. There's also a 4th ed available. An encyclopedia of nearly everything you need to know, but not particularly light-reading material. Numerical Linear Algebra by Trefethen and Bau. Easier to read book with many useful exercises and a more 'conversational' tone. A First Course in Numerical Methods by Chen Greif and Uri Ascher. Broader focus than just numerical linear algebra, but good for first-time exposure to computational aspects of linear algebra. 3/47 Ron Estrin Lecture 1: Preliminaries Introduction A little about me: I'm a third-year ICME PhD student working in linear algebra and optimization. Email: [email protected]. Webpage: http://stanford.edu/~restrin. Let's begin! 4/47 Ron Estrin Lecture 1: Preliminaries Vector Spaces Definition A vector space is a set V and field F with a binary operation addition (u + v = w 2 V for all u; v 2 V ), and scalar multiplication (αu = v 2 V for all u 2 V , α 2 F) such that the following axioms hold: Commutativity: u + v = v + u Associativity: (u + v) + w = u + (v + w) Additive identity: There exists 0 2 V s.t. v + 0 = v, 8v 2 V Additive inverse: 8v 2 V there exists w 2 V s.t. v + w = 0 Multiplicative identity: There exists 1 2 F s.t. 1v = v, 8v 2 V Distributativity: α(u + v) = αu + αv and (α + β)v = αv + βv 5/47 Ron Estrin Lecture 1: Preliminaries Subspaces Definition A subspace U of a vector space V (with the field F) is a subset U ⊆ V such that 0 2 U and Closed under addition: u + v 2 U for all u; v 2 U Closed under scalar multiplication: αv 2 U for all v 2 U Important: A subspace is itself a vector space. 6/47 Ron Estrin Lecture 1: Preliminaries Examples of vector spaces n n Euclidean space: (Everyone's favourite) R or C (columns of numbers). Example subspace: Choose set αi 2 R, then n Pn U = fx 2 R j i=n αi xi = 0g is a subspace. n n Generally we'll discuss Euclidean space with either R or C . Continuous real-valued functions on [0,1]. Example subspaces: Polynomials of degree ≤ n (Pn(x)), U = ff 2 C[0; 1] j f (0) = f (1) = 0g. 7/47 Ron Estrin Lecture 1: Preliminaries Span of vectors Definition The span of a set of vectors is the subspace of all linear linear combinations of those vectors ( n ) X span fv1;:::; vk g = w j w = αi vi : i=1 Examples: 80 1 0 19 80 1 9 < 1 1 = < α1 = span @0A ; @ 0 A = @ 0 A j α1; α2 2 R : : 1 −1 ; : α2 ; n 2k o span fx j k 2 Ng = fPolynomials with even degree termsg 8/47 Ron Estrin Lecture 1: Preliminaries Linear independence/dependence Definition n A set of vectors fvi gi=1 is linearly independent if n X αi vi = 0 =) αi = 0; i = 1;:::; n: i=1 Otherwise, the set is linearly dependent. Linearly dependent sets are redundant, since we can represent any vector (if αj 6= 0) as 1 X vj = αi vi : αj i6=j 9/47 Ron Estrin Lecture 1: Preliminaries Linear independence/dependence Examples: 80 1 0 1 0 19 < 1 1 2 = The set fv1; v2; v3g = @2A ; @ 1 A ; @3A is linearly : 1 −1 0 ; dependent since v1 + v2 − v3 = 0. 80 1 0 1 0 19 < 2 1 2 = The set fv1; v2; v3g = @2A ; @ 1 A ; @3A is linearly : 1 −1 0 ; independent. 10/47 Ron Estrin Lecture 1: Preliminaries Bases Definition n A set of vectors fvi gi=1 generates a vector space U if span fvi g = U. Definition n A set of vectors fvi gi=1 is a basis for a vector space U if n span fvi g = U and the set fvi gi=1 is linearly independent. n With a basis, we can express any u 2 U in the basis fvi gi=1 as n X u = αi vi ; i=1 for some coefficients αi . 11/47 Ron Estrin Lecture 1: Preliminaries Dimension of a vector space Definition The dimension of a vector space V is the number of vectors in any fixed basis of V , dim(V ) = jvectors in basis of V j : Remember: The dimension depends only on the vector space, not on the basis! Not all vector spaces are finite dimensional (e.g. space of continuous functions), but for numerical linear algebra, we'll generally only care about the finite-dimensional ones. 12/47 Ron Estrin Lecture 1: Preliminaries Example bases 2 n One basis for Pn(x) is the set of monomials f1; x; x ;:::; x g. Another basis for the same space is the set of Chebyshev polynomials of the first kind, fP0; P1;:::; Png: P0(x) = 1 P1(x) = x Pn(x) = 2xPn−1(x) − Pn−2(x): In both cases, the cardinality of the basis sets is n + 1, so the dimension of the space is n + 1. Although they span the same space, these bases have very different properties! 13/47 Ron Estrin Lecture 1: Preliminaries Inner product space Definition An inner product space is a vector space V with a defined inner product h·; ·i : V × V ! F; such that the following properties hold: Conjugate symmetry: hu; vi = hv; ui Linearity in first argument: hαu + v; wi = α hu; wi + hv; wi : Positive-Definiteness: hu; ui ≥ 0 with equality iff u = 0. Formal definition for `products of vectors'. 14/47 Ron Estrin Lecture 1: Preliminaries Inner product space Examples: n Dot-product for C : (Everyone's favourite) Defined as n X ∗ hu; vi = u¯i vi = v u: i=1 Also known as the `2 inner product. L2 inner product for functions on [0; 1]. Defined as Z 1 hf ; giL2 = f (x)g(x)dx: 0 15/47 Ron Estrin Lecture 1: Preliminaries Norms Definition A norm on a vector space V is a function k·k : V ! R+ [ 0 such that the following properties hold: Absolute homogeneity: kαvk = jαj kvk for all α 2 F and v 2 V Sub-additivity (triangle inequality): ku + vk ≤ kuk + kvk Nondegeneracy: kvk = 0 iff v = 0 Norms generalize the idea \length" of vectors. All norms are convex functions. 16/47 Ron Estrin Lecture 1: Preliminaries Norms Examples: Euclidean norm: Defined as 1 n ! 2 X 2 ∗ kuk2 = jui j = u u: i=1 Also called the `2-norm. An example of a norm defined by an inner product. `p-norm: Defined as 1 n ! p X p kukp = jui j : i=1 `1-norm: Define as kuk1 = max jui j: 1≤i≤n 17/47 Ron Estrin Lecture 1: Preliminaries Euclidean inner product and geometry Let u and v be two vectors, with angle θ in between. The Euclidean norm is exactly the usual notion of `length' of a vector, and the inner product satisfies hu; vi2 = kuk2 kvk2 cos θ: 18/47 Ron Estrin Lecture 1: Preliminaries Exercises 1 Prove that every inner product defines a norm. That is, show that 1 kuk = (hu; ui) 2 ; is a norm. 2 Prove the cosine law. If a, b, c are the sides of the triangle, and θ is the angle between a and b, then jcj2 = jaj2 + jbj2 − 2jajjbj cos θ: 19/47 Ron Estrin Lecture 1: Preliminaries Important inequalities Triangle Inequality: ku + vk ≤ kuk + kvk : Reverse Triangle Inequality: ku − vk ≥ jkuk − kvkj : 20/47 Ron Estrin Lecture 1: Preliminaries Recall that for the Euclidean inner product, hu; vi2 = kuk2 kvk2 cos θ: Since 0 ≤ j cos θj ≤ 1, Cauchy-Schwarz clearly holds, and we can observe when sharpness occurs: when θ = 0. Important inequalities Cauchy-Schwarz Inequality: Let the norm k·k be induced by the inner product h·; ·i. Then j hu; vi j ≤ kuk kvk : It basically says that the size of the inner product is bounded by the product of the size of the vectors themselves. 21/47 Ron Estrin Lecture 1: Preliminaries Important inequalities Cauchy-Schwarz Inequality: Let the norm k·k be induced by the inner product h·; ·i. Then j hu; vi j ≤ kuk kvk : It basically says that the size of the inner product is bounded by the product of the size of the vectors themselves. Recall that for the Euclidean inner product, hu; vi2 = kuk2 kvk2 cos θ: Since 0 ≤ j cos θj ≤ 1, Cauchy-Schwarz clearly holds, and we can observe when sharpness occurs: when θ = 0. 21/47 Ron Estrin Lecture 1: Preliminaries Definition Vectors u, v are orthogonal with respect to an inner product if hu; vi = 0: For the Euclidean inner product, this is the usual notion of orthogonality, i.e. two vectors are orthogonal if θ = π=2 since hu; vi2 = kuk2 kvk2 cos θ: 22/47 Ron Estrin Lecture 1: Preliminaries Definition n An orthogonal basis is a basis fvi gi=1 such that hvi ; vj i = 0 for i 6= j.