Orthogonal Matrices, Change of Basis, Rank

Home , Change of basis, Orthogonal matrix, Orthonormal basis

Orthogonal matrices, change of basis, rank

Math Tools for Neuroscience (NEU 314) Spring 2016

Jonathan Pillow Princeton Neuroscience Institute & Psychology.

Lecture 6 (Thursday 2/18) accompanying notes/slides today’s topics

• orthonormal basis • change of basis • orthogonal matrix • rank • column space and row space • null space is clearly a vector space [verify].

Working backwards, a set of vectors is said to span a vector space if one can write any v vector in the vector space as a linear com- 1 v3 bination of the set. A spanning set can be redundant: For example, if two of the vectors are identical, or are scaled copies of each other. This redundancy is formalized by deﬁning linear independence.Asetofvec- tors {⃗v1,⃗v2,...⃗vM } is linearly independent if v2 (and only if) the only solution to the equation

αn⃗vn =0 !n is αn =0(for all n). basis A basis for a vector space is a linearly independent spanning set. For example, con- • set of vectors that can “span” (form via linear combination) all points in a vector space sider the plane of this page. One vector is not enough to span the plane. Scalar multi- v ples of this vector will trace out a line (which v v v 2 is a subspace), but cannot “get off the line” vv 2 1 v v to cover the rest of the plane. But two vec- 1 v v tors are sufﬁcient to span the entire plane. 1 Bases are not unique: any two vectors will do, as long as they don’t lie along the same 1D vector space Two diﬀerent (orthonormal) line. Three vectors are redundant: one can (subspace of R2) bases for the same 2D always be written as a linear combination of vector space the other two. In general, the vector space N R requires a basis of size N.

e Geometrically, the basis vectors deﬁne a set of coordinate axes for the space (although e ˆ2 they need not be perpendicular). The stan- e dard basis is the set of unit vectors that lie along the axes of the space:

1 0 0 eˆ1 ⎛ 0 ⎞ ⎛ 1 ⎞ ⎛ 0 ⎞ eˆ1 = ⎜ 0 ⎟, eˆ2 = ⎜ 0 ⎟,...eˆN = ⎜ 0 ⎟. ⎜ . ⎟ ⎜ . ⎟ ⎜ . ⎟ ⎜ . ⎟ ⎜ . ⎟ ⎜ . ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ v ⎜ 0 ⎟ ⎜ 0 ⎟ ⎜ 1 ⎟ S x ⎝ ⎠ ⎝ ⎠ ⎝ ⎠ 5 x S e ) v x e x S x S e

x e v x S e S x) v x e x S x S e

x e

o o o o o

3 orthonormal basis

• basis composed of orthogonal unit vectors Change of basis • Let B denote a matrix whose columns form an orthonormal basis for a vector space W

If B is full rank (n x n), then

we can get back to the original basis through multiplication by B Change of basis • Let B denote a matrix whose columns form an orthonormal basis for a vector space W

Vector of projections of v along each basis vector Orthogonal matrix • In this case (full rank, orthogonal columns), B is an orthogonal matrix

Properties:

length- preserving nothing. This matrix is called the identity,denotedI.

If an element of the diagonal is zero, then the associated axis is annihilated. The set of vectors that are annihilated by the matrix form a vector space [prove], which is called the row nullspace,orsimplythenullspace of the matrix. v v 2 0 2 [ 0 0 ] v1 2v1

Another implication of a zero diagonal element is that the matrix cannot “reach” the entire output space, but only a proper subspace. This space is called the column space of the matrix, since it is spanned by the matrix columns. The rank of a matrix is just the dimensionality of the column space. A matrix is said to have full rankOrthogonalif its rank is equal to matrix the smaller of its two dimensions. An orthogonal matrix is a square• matrix2D example: rotation matrix whose columns are pairwise orthogonal unit vectors. Remember that the columns of a matrix describe the response of the system to e^ the standard basis. Thus an orthogonal ma- 2 ( Ο ) = trix maps the standard basis onto a new set e^ 1 ^ of N orthogonal axes, which form an alter- ^ Ο(e1 ) Ο (e2 ) native basis for the space. This operation is cos θ sin θ a generalized rotation, since it corresponds to e g .. Ο = sin θ cosθ] a physical rotation of the space and possibly [ negation of some axes. Thus, the product of two orthogonal matrices is also orthogonal. Note that an orthogonal is full rank (it has no nullspace), since a rotation cannot annihilate any non-zero vector.

Linear Systems of Equations

The classic motivation for the study of linear algebra is the solution of sets of linear equations such as a11v1 + a12v2 + ...+ a1N vN = b1 a21v1 + a22v2 + ...+ a2N vN = b2 . . aM1v1 + aM2v2 + ...+ aMNvN = bM

8 Rank • the rank of a matrix is equal to • # of linearly independent columns • # of linearly independent rows (remarkably, these are always the same) equivalent deﬁnition: • the rank of a matrix is the dimensionality of the vector space spanned by its rows or its columns for an m x n matrix A: rank(A) ≤ min(m,n)

(can’t be greater than # of rows or # of columns) column space of a matrix W:

n × m matrix

vector space spanned by the columns of W c1 … cm

• these vectors live in an n-dimensional space, so the column space is a subspace of Rn row space of a matrix W:

n × m matrix

vector space spanned by the … rows of W

• these vectors live in an m-dimensional space, so the column space is a subspace of Rm null space of a matrix W: n × m matrix

• the vector space consisting of r1 all vectors that are orthogonal to … the rows of W rn

• equivalently: the null space of W is the vector space of all vectors x such that Wx = 0.

• the null space is therefore entirely orthogonal to the row space of a matrix. Together, they make up all of Rm. is clearly a vector space [verify].

αn⃗vn =0 n ! null space of a matrix W: is αn =0(for all n). W = ( v 1 ) A basis for a vector space is a linearly independent spanning set. For example, con-null space sider the plane of this page. One vector is 1D vector space not enough to span the plane. Scalar multi- spanned by v1 v ples of this vector will trace out a line (which v v v 2 is a subspace), but cannot “get off the line” vv 2 basis for null space 1 v v to cover the rest of the plane. But two vec- 1 v v tors are sufﬁcient to span the entire plane. 1 Bases are not unique: any two vectors will do, as long as they don’t lie along the same line. Three vectors are redundant: one can always be written as a linear combination of the other two. In general, the vector space N R requires a basis of size N.

x e v x S e S x) v x e x S x S e

x e

o o o o o