Linear Algebra III: vector spaces Math Tools for Neuroscience (NEU 314) Fall 2016 Jonathan Pillow Princeton Neuroscience Institute & Psychology. Lecture 4 (Tuesday 9/27) accompanying notes/slides Outline Last time: • linear combination • linear independence / dependence • matrix operations: transpose, multiplication, inverse Topics: • matrix equations • vector space, subspace • basis, orthonormal basis • orthogonal matrix • rank • row space / column space • null space • change of basis inverse • If A is a square matrix, its inverse A-1 (if it exists) satisfies: “the identity” (eg., for 4 x 4) The identity matrix for any vector “the identity” (eg., for 4 x 4) two weird tricks • transpose of a product • inverse of a product (Square) Matrix Equation assume (for now) square and invertible left-multiply both sides by inverse of A: is clearly a vector space [verify]. Working backwards, a set of vectors is said to span a vector space if one can write any v vector in the vector space as a linear com- 1 v3 bination of the set. A spanning set can be redundant: For example, if two of the vec- tors are identical, or are scaled copies of each other. This redundancy is formalized by defining linear independence.Asetofvec- tors {⃗v1,⃗v2,...⃗vM } is linearly independent if v2 (and only if) the only solution to the equation αn⃗vn =0 !n is αn =0(for all n). vector space & basis • vector space - set of all points that can be obtained by A basis for a vector space is a linearly in- linear combinations some set of vectors dependent spanning set. For example, con- • basis - a set of linearly independent vectors that generate sider the plane of this page. One vector is (through linear combinations) all points in a vector space not enough to span the plane. Scalar multi- v ples of this vector will trace out a line (which v v v 2 is a subspace), but cannot “get off the line” vv 2 1 v v to cover the rest of the plane. But two vec- 1 v v tors are sufficient to span the entire plane. 1 Bases are not unique: any two vectors will do, as long as they don’t lie along the same 1D vector space Two different bases for the line. Three vectors are redundant: one can (subspace of R2) same 2D vector space always be written as a linear combination of the other two. In general, the vector space N R requires a basis of size N. e Geometrically, the basis vectors define a set of coordinate axes for the space (although e ˆ2 they need not be perpendicular). The stan- e dard basis is the set of unit vectors that lie along the axes of the space: 1 0 0 eˆ1 ⎛ 0 ⎞ ⎛ 1 ⎞ ⎛ 0 ⎞ eˆ1 = ⎜ 0 ⎟, eˆ2 = ⎜ 0 ⎟,...eˆN = ⎜ 0 ⎟. ⎜ . ⎟ ⎜ . ⎟ ⎜ . ⎟ ⎜ . ⎟ ⎜ . ⎟ ⎜ . ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ v ⎜ 0 ⎟ ⎜ 0 ⎟ ⎜ 1 ⎟ S x ⎝ ⎠ ⎝ ⎠ ⎝ ⎠ 5 x S e ) v x e x S x S e x e v x S e S x) v x e x S x S e x e o o o o o 3 o o o o o 3 is clearly a vector space [verify]. Working backwards, a set of vectors is said to span a vector space if one can write any v vector in the vector space as a linear com- 1 v3 bination of the set. A spanning set can be redundant: For example, if two of the vec- tors are identical, or are scaled copies of each other. This redundancy is formalized by defining linear independence.Asetofvec- tors {⃗v1,⃗v2,...⃗vM } is linearly independent if v2 (and only if) the only solution to the equation αn⃗vn =0 !n is αn =0(for all n). span - to generate via linear combination • vector space - set of all points that can be spanned A basis for a vector space is a linearly in- by some set of vectors dependent spanning set. For example, con- • basis - a set of vectors that can span a vector space sider the plane of this page. One vector is not enough to span the plane. Scalar multi- v ples of this vector will trace out a line (which v v v 2 is a subspace), but cannot “get off the line” vv 2 1 v v to cover the rest of the plane. But two vec- 1 v v tors are sufficient to span the entire plane. 1 Bases are not unique: any two vectors will do, as long as they don’t lie along the same 1D vector space Two different bases for the line. Three vectors are redundant: one can (subspace of R2) same 2D vector space always be written as a linear combination of the other two. In general, the vector space N R requires a basis of size N. e Geometrically, the basis vectors define a set of coordinate axes for the space (although e ˆ2 they need not be perpendicular). The stan- e dard basis is the set of unit vectors that lie along the axes of the space: 1 0 0 eˆ1 ⎛ 0 ⎞ ⎛ 1 ⎞ ⎛ 0 ⎞ eˆ1 = ⎜ 0 ⎟, eˆ2 = ⎜ 0 ⎟,...eˆN = ⎜ 0 ⎟. ⎜ . ⎟ ⎜ . ⎟ ⎜ . ⎟ ⎜ . ⎟ ⎜ . ⎟ ⎜ . ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ v ⎜ 0 ⎟ ⎜ 0 ⎟ ⎜ 1 ⎟ S x ⎝ ⎠ ⎝ ⎠ ⎝ ⎠ 5 x S e ) v x e x S x S e x e v x S e S x) v x e x S x S e x e o o o o o 3 o o o o o 3 is clearly a vector space [verify]. Working backwards, a set of vectors is said to span a vector space if one can write any v vector in the vector space as a linear com- 1 v3 bination of the set. A spanning set can be redundant: For example, if two of the vec- tors are identical, or are scaled copies of each other. This redundancy is formalized by defining linear independence.Asetofvec- tors {⃗v1,⃗v2,...⃗vM } is linearly independent if v2 (and only if) the only solution to the equation αn⃗vn =0 !n is αn =0(for all n). orthonormal basis A basis for a vector space is a linearly in- • basis composed of orthogonal unit vectors dependent spanning set. For example, con- sider the plane of this page. One vector is not enough to span the plane. Scalar multi- v ples of this vector will trace out a line (which v v v 2 is a subspace), but cannot “get off the line” vv 2 1 v v to cover the rest of the plane. But two vec- 1 v v tors are sufficient to span the entire plane. 1 Bases are not unique: any two vectors will do, as long as they don’t lie along the same • Two different orthonormal bases line. Three vectors are redundant: one can for the same vector space always be written as a linear combination of the other two. In general, the vector space N R requires a basis of size N. e Geometrically, the basis vectors define a set of coordinate axes for the space (although e ˆ2 they need not be perpendicular). The stan- e dard basis is the set of unit vectors that lie along the axes of the space: 1 0 0 eˆ1 ⎛ 0 ⎞ ⎛ 1 ⎞ ⎛ 0 ⎞ eˆ1 = ⎜ 0 ⎟, eˆ2 = ⎜ 0 ⎟,...eˆN = ⎜ 0 ⎟. ⎜ . ⎟ ⎜ . ⎟ ⎜ . ⎟ ⎜ . ⎟ ⎜ . ⎟ ⎜ . ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ v ⎜ 0 ⎟ ⎜ 0 ⎟ ⎜ 1 ⎟ S x ⎝ ⎠ ⎝ ⎠ ⎝ ⎠ 5 x S e ) v x e x S x S e x e v x S e S x) v x e x S x S e x e o o o o o 3 o o o o o 3 Orthogonal matrix • Square matrix whose columns (and rows) form an orthonormal basis (i.e., are orthogonal unit vectors) Properties: length- preserving nothing. This matrix is called the identity,denotedI. If an element of the diagonal is zero, then the associated axis is annihilated. The set of vectors that are annihilated by the matrix form a vector space [prove], which is called the row nullspace,orsimplythenullspace of the matrix. v v 2 0 2 [ 0 0 ] v1 2v1 Another implication of a zero diagonal element is that the matrix cannot “reach” the entire output space, but only a proper subspace. This space is called the column space of the matrix, since it is spanned by the matrix columns. The rank of a matrix is just the dimensionality of the column space. A matrix is said to have full rankOrthogonalif its rank is equal to matrix the smaller of its two dimensions. An orthogonal matrix is a square• matrix2D example: rotation matrix whose columns are pairwise orthogonal unit vectors. Remember that the columns of a matrix describe the response of the system to e^ the standard basis. Thus an orthogonal ma- 2 ( Ο ) = trix maps the standard basis onto a new set e^ 1 ^ of N orthogonal axes, which form an alter- ^ Ο(e1 ) Ο (e2 ) native basis for the space. This operation is cos θ sin θ a generalized rotation, since it corresponds to e g .. Ο = sin θ cosθ] a physical rotation of the space and possibly [ negation of some axes. Thus, the product of two orthogonal matrices is also orthogonal. Note that an orthogonal is full rank (it has no nullspace), since a rotation cannot annihilate any non-zero vector. Linear Systems of Equations The classic motivation for the study of linear algebra is the solution of sets of linear equations such as a11v1 + a12v2 + ...+ a1N vN = b1 a21v1 + a22v2 + ...+ a2N vN = b2 . aM1v1 + aM2v2 + ...+ aMNvN = bM 8 Rank • the rank of a matrix is equal to • # of linearly independent columns • # of linearly independent rows (remarkably, these are always the same) equivalent definition: • the rank of a matrix is the dimensionality of the vector space spanned by its rows or its columns for an m x n matrix A: rank(A) ≤ min(m,n) (can’t be greater than # of rows or # of columns) column space of a matrix W: n × m matrix vector space spanned by the columns of W c1 … cm • these vectors live in an n-dimensional space, so the column space is a subspace of Rn row space of a matrix W: n × m matrix r1 vector space spanned by the … rows of W rn • these vectors live in an m-dimensional space, so the column space is a subspace of Rm null space of a matrix W: n × m matrix • the vector space consisting of r1 all vectors that are orthogonal to … the rows of W rn • equivalently: the null space of W is the vector space of all vectors x such that Wx = 0.