Vector spaces

Paul Schrimpf

Normed vector spaces Examples

Inner product spaces Useful inequalities Vector spaces Projections

Row, column, and null space Row space Paul Schrimpf Column space Null space

Applications UBC Portfolio analysis Economics 526 First and second welfare theorems Lines, planes, and hyperplanes September 23, 2013 Vector spaces Paul Schrimpf Overview

Normed vector spaces Examples

Inner product spaces Useful inequalities

Projections

Row, column, and null space 3 n Row space • Main idea: take intuition from R and apply to R and Column space Null space other vector spaces Applications Portfolio analysis First and second welfare theorems Lines, planes, and hyperplanes Vector spaces

Paul Schrimpf 1. Normed vector spaces Normed Examples vector spaces Examples Inner product 2. Inner product spaces spaces Useful inequalities Useful inequalities Projections Row, column, 3. and null space Projections Row space Column space Null space 4. Row, column, and null space Applications Row space Portfolio analysis First and second welfare theorems Column space Lines, planes, and hyperplanes Null space

5. Applications Portfolio analysis First and second welfare theorems Vector spaces

Paul Schrimpf

Normed vector spaces Examples

Inner product spaces Useful inequalities

Projections Section 1

Row, column, and null space Row space Column space Normed vector spaces Null space

Applications Portfolio analysis First and second welfare theorems Lines, planes, and hyperplanes Vector spaces Paul Schrimpf Normed vector spaces

Normed vector spaces • Measure length or distance Examples

Inner product spaces Definition Useful inequalities F · ∥·∥ Projections A normed vector space, (V, , +, , ), is a vector space F Row, column, with a function, called the norm, from V to and denoted by and null space ∥ ∥ Row space v with the following properties: Column space Null space 1. ∥v∥ ≥ 0 and ∥v∥ = 0 iff v = 0, Applications ∥ ∥ | | ∥ ∥ ∈ F Portfolio analysis 2. αv = α v for all α , First and second welfare theorems 3. The triangle inequality holds: Lines, planes, and hyperplanes

∥v1 + v2∥ ≤ ∥v1∥ + ∥v2∥

for all v1, v2 ∈ V.

• Shortest distance between two points is a straight line Vector spaces Paul Schrimpf Examples

Normed vector spaces Examples Inner product Example spaces 3 Useful inequalities R is a normed vector space with norm Projections √ Row, column, ∥ ∥ 2 2 2 and null space x = x1 + x2 + x3 . Row space Column space Null space This norm is exactly how we usually measure distance. For Applications Portfolio analysis this reason, it is called the Euclidean norm. First and second n welfare theorems More generally, for any n, R , is a normed vector space with Lines, planes, and hyperplanes norm v u u∑n ∥ ∥ t 2 x = xi . i=1 Vector spaces Paul Schrimpf Examples

Normed vector spaces Examples

Inner product spaces Useful inequalities Example Projections Rn Row, column, with the norm and null space ( ) Row space p 1/p Column space ∑ Null space ∥ ∥ | |p x p = xi Applications i=1 Portfolio analysis First and second welfare theorems 1 Lines, planes, and for p ∈ [1, ∞] is a normed vector space. This norm is called hyperplanes the p-norm.

1 Where ∥x∥∞ = max1≤i≤n |xi | Vector spaces Paul Schrimpf Examples

Normed vector spaces Examples

Inner product spaces Useful inequalities Example Projections Lp(0, 1) with p-norm Row, column, and null space ( ) ∫ 1/p Row space 1 Column space p Null space ∥ ∥ | | f p = f(x) dx Applications 0 Portfolio analysis First and second welfare theorems Lp Lines, planes, and is a normed vector space. Moreover, (0, 1) is a different hyperplanes space for different p. For example, 1 ̸∈ Lp(0, 1), but x1/p 1 ∈ Lq(0, 1) for q < p. x1/p Vector spaces

Paul Schrimpf

Normed vector spaces Examples

Inner product spaces Useful inequalities

Projections Section 2

Row, column, and null space Row space Column space Inner product spaces Null space

Applications Portfolio analysis First and second welfare theorems Lines, planes, and hyperplanes Vector spaces Paul Schrimpf Inner product spaces

Normed vector spaces • Measure angles Examples

Inner product spaces Definition Useful inequalities A real inner product space is a vector space over the field Projections R with an additional operation called the inner product that Row, column, and null space is function from V × V to R. We denote the inner product of Row space ∈ ⟨ ⟩ Column space v1, v2 V by v1, v2 . It has the following properties: Null space ⟨ ⟩ ⟨ ⟩ Applications 1. Symmetry: v1, v2 = v2, v1 Portfolio analysis First and second 2. Linear: ⟨av + bv , v ⟩ = a ⟨v , v ⟩ + b ⟨v , v ⟩ for welfare theorems 1 2 3 1 3 2 3 Lines, planes, and ∈ R hyperplanes a, b 3. Positive definite: ⟨v, v⟩ ≥ 0 and equals 0 iff v = 0.

• Inner product space is also a normed vector space √ ∥x∥ = ⟨x, x⟩. Vector spaces Paul Schrimpf Example

Normed vector spaces Examples

Inner product spaces • Rn Useful inequalities with the dot product,

Projections ∑n Row, column, T and null space ⟨x, y⟩ = x · y = xi yi = x y Row space Column space i=1 Null space Applications • Norm induced by the inner product is the Euclidean Portfolio analysis First and second norm v welfare theorems u Lines, planes, and n hyperplanes √ u∑ ∥ ∥ ⟨ ⟩ t 2 x = x, x = xi i=1 Vector spaces Paul Schrimpf Measuring angles

Normed vector spaces Examples • Inner product spaces 2 Useful inequalities ∥x + y∥ = ⟨x + y, x + y⟩ Projections = ⟨x, x⟩ + 2 ⟨x, y⟩ + ⟨y, y⟩ Row, column, and null space Row space • Rn Column space In with the Euclidean norm when x and y are at right Null space angles to one other, ⟨x, y⟩ = 0, and we have the Applications Portfolio analysis Pythagorean theorem: First and second welfare theorems Lines, planes, and 2 2 2 hyperplanes ∥x + y∥ = ∥x∥ + ∥y∥ .

Definition Let x, y ∈ V, an inner product space. x and y are orthogonal iff ⟨x, y⟩ = 0. Vector spaces Paul Schrimpf Measuring angles

Normed Theorem vector spaces n Examples Let u, v ∈ R , then the angle between them is Inner product spaces − ⟨u, v⟩ Useful inequalities θ = cos 1 . Projections ∥u∥ ∥v∥ Row, column, and null space Row space Column space Proof. Null space Draw picture. Applications ∥ ∥ Portfolio analysis tv First and second cos θ = . welfare theorems ∥u∥ Lines, planes, and hyperplanes Use Pythagorean theorem,

∥u∥2 = ∥tv∥2 + ∥u − tv∥2 ∥u∥2 =t2 ∥v∥2 + ∥u∥2 − 2t ⟨u, v⟩ + t2 ∥v∥2 2t ⟨u, v⟩ =2t2 ∥v∥2 Vector spaces Paul Schrimpf Useful inequalities

Normed • Triangle inequality vector spaces Examples Theorem (Reverse triangle inequality) Inner product spaces Let V be a normed vector space and x, y ∈ V. Then Useful inequalities

Projections |∥x∥ − ∥y∥| ≤ ∥x − y∥ . Row, column, and null space Row space Column space Null space Proof. Applications By the usual triangle inequality, Portfolio analysis First and second welfare theorems Lines, planes, and ∥x∥ + ∥x − y∥ ≥ ∥y∥ hyperplanes ∥x − y∥ ≥ ∥y∥ − ∥x∥

and

∥y∥ + ∥y − x∥ ≥ ∥x∥ ∥y − x∥ ≥ ∥x∥ − ∥y∥ . Vector spaces Paul Schrimpf Useful inequalities

Normed vector spaces Theorem (Cauchy-Schwarz inequality ) Examples ∈ Inner product Let V be an inner product space and let u, v V. Then, spaces Useful inequalities |⟨ ⟩| ≤ ∥ ∥ ∥ ∥ Projections u, v u v .

Row, column, and null space Row space Column space Proof. Null space ⟨u,v⟩ Setup as before, we can show that t = 2 . Now, let Applications ∥v∥ Portfolio analysis z = u − tv. By the Pythagorean theorem, First and second welfare theorems Lines, planes, and 2 2 2 hyperplanes ∥u∥ = ∥tv∥ + ∥z∥ ⟨u, v⟩2 = + ∥z∥2 ∥v∥2

⟨ ⟩2 ∥z∥2 ≥ 0, so ∥u∥2 ≥ u,v ∥u∥ ∥v∥ ≥ | ⟨u, v⟩ |. ∥v∥2 Vector spaces

Paul Schrimpf

Normed vector spaces Examples

Inner product spaces Useful inequalities

Projections Section 3

Row, column, and null space Row space Column space Projections Null space

Applications Portfolio analysis First and second welfare theorems Lines, planes, and hyperplanes Vector spaces Paul Schrimpf Projections

Normed vector spaces Examples

Inner product Definition spaces Let V be an inner product space and x, y ∈ V. The Useful inequalities

Projections projection of y onto x is

Row, column, and null space ⟨y, x⟩ Row space Pxy = x. Column space ∥ ∥2 Null space x

Applications Portfolio analysis More generally, the projection of y onto a finite set First and second welfare theorems {x , x , ..., x } is Lines, planes, and 1 2 k hyperplanes ∑k ∑ P k y = P j−1 y. {xj }j =1 − xj i=1 Pxi xj j=1 Vector spaces Paul Schrimpf Projections

Normed vector spaces Examples

Inner product spaces Definition Useful inequalities More generally still, if X ⊆ V is a linear subspace, then the Projections projection of y onto X is Row, column, and null space Row space PX y = P{ }k y Column space bj j=1 Null space Applications ∈ Portfolio analysis where bj X and b1, ..., bk span X. First and second welfare theorems Finally, if Y ⊆ V the projection of Y onto X is just the set Lines, planes, and hyperplanes consisting of the projection of each element of y onto X, i.e.

PX Y = {PX y : y ∈ Y}. Vector spaces Paul Schrimpf Projections

Normed Lemma vector spaces Examples Any projection is an idempotent linear transformation. Inner product spaces Proof. Useful inequalities Projections • Verify that projections have the two properties required Row, column, for them to be linear transformations. and null space Row space • Column space Show that projections are idempotent. Null space ( ) Applications ⟨x, y⟩ Portfolio analysis P (P y) =P x First and second x x x 2 welfare theorems ∥x∥ Lines, planes, and hyperplanes ⟨ ⟩ ⟨x,y⟩ x, ∥ ∥2 x = x x ∥x∥2 ⟨x, y⟩ ⟨x, x⟩ = x ∥x∥2 ∥x∥2

=Pxy. Vector spaces

Paul Schrimpf

Normed vector spaces Examples

Inner product spaces Useful inequalities

Projections Section 4

Row, column, and null space Row space Column space Row, column, and null space Null space

Applications Portfolio analysis First and second welfare theorems Lines, planes, and hyperplanes Vector spaces Paul Schrimpf . Row space

Normed Definition vector spaces Let A be an m by n matrix. The row space of A, denoted Examples Row(A), is the space spanned by the row vectors of A. Inner product spaces n Useful inequalities • Row(A) ⊆ R Projections

Row, column, Lemma and null space Performing Gaussian elimination does not change the row Row space Column space space of a matrix. Null space

Applications Proof. Portfolio analysis Let a1, ..., am be the row vectors of A. Each step of First and second welfare theorems Gaussian elimination transforms some aj into aj + gak with Lines, planes, and hyperplanes k ≠ j or g ≠ −1. Can show that

span(a1, ..., am) = span(a1 + gak, ..., am).

Corollary The dimension of the row space of a matrix is equal to its rank. Vector spaces Paul Schrimpf Column space

Normed vector spaces Examples

Inner product spaces Useful inequalities Definition Projections

Row, column, Let A be an m by n matrix. The column space of A, denoted and null space Col(A), is the space spanned by the column vectors of A. Row space Column space Null space • ColA ⊆ Rm Applications Portfolio analysis First and second welfare theorems Lemma Lines, planes, and hyperplanes Let A be an m by n matrix. Then Ax = b has a solution iff b ∈ Col(A). Vector spaces

Paul Schrimpf

Normed Definition vector spaces Examples A column of a matrix, A, is basic if the corresponding Inner product column of the row echelon form, Ar , contains a pivot. spaces Useful inequalities Theorem Projections Col( ) Row, column, The basic columns of A form a basis for A . and null space Row space Column space Proof. Null space Let A be m × n and denote its columns as v1, ..., vn. Let Ar Applications Portfolio analysis be the row echelon form of A and denotes its columns as First and second welfare theorems w1, ..., wn. Let wi1 , ..., wik be the basic columns of Ar . Each Lines, planes, and hyperplanes has more zeros, so wi1 , ..., wik are linearly independent. By definition of row echelon form, the final m − k rows of Ar are ≤ all zero. Therefore dimCol(Ar ) k, and wi1 , ..., wik must be a basis for Col(Ar ). Vector spaces

Paul Schrimpf Continued. Normed vector spaces Now we show that vi , ..., vi are a basis for Col(A). Suppose Examples 1 k

Inner product spaces c1vi1 + ... + ckvik = 0. Useful inequalities Projections Then we could do Gaussian elimination to convert this Row, column, and null space system to Row space Column space c1wi1 + ... + ckwik = 0. Null space

Applications wi1 , ..., wik are linearly independent so c1 = 0, ...ck = 0. Portfolio analysis ̸∈ { } First and second Add any other vj , j i1, ..., ik , then by the same argument welfare theorems Lines, planes, and there must exist a non-zero c than solves hyperplanes

c1vi1 + ... + ckvik + cjvj = 0.

Thus, vi1 , ..., vik is a basis for Col(A). Vector spaces

Paul Schrimpf

Normed vector spaces Examples

Inner product spaces Useful inequalities Corollary Projections The dimensions of the row and column spaces of any matrix Row, column, and null space are equal. Row space Column space Null space Corollary Applications T Portfolio analysis rankA = rankA . First and second welfare theorems Lines, planes, and hyperplanes Vector spaces Paul Schrimpf Null space

Normed vector spaces Examples

Inner product Definition spaces Let A be m by n. The set of solutions to the homogeneous Useful inequalities

Projections equation Ax = 0 is the null space (or kernel) of A, denoted N Row, column, by (A) (or Null(A)). and null space Row space Column space Definition Null space Let V ⊆ Rn be a linear subspace, and let c ∈ Rn be a fixed Applications Portfolio analysis vector. The set First and second welfare theorems Lines, planes, and n hyperplanes {x ∈ R : x = v + c for some v ∈ V}

is called the set of translates of V by c, and is denoted c + V. Any set of translates of a linear subspace is called an affine space. Vector spaces Paul Schrimpf Lemma Normed Let Ax = b be an m by n system of linear equations. Let x0 vector spaces Examples be any particular solution. Then the set of solutions is Inner product x0 + N (A). spaces Useful inequalities

Projections Proof. ∈ N Row, column, Let w x0 + (A). Then and null space Row space Column space Aw =A(x0) + A(w − x0) Null space | {z } Applications ∈N (A) Portfolio analysis First and second welfare theorems =b + 0. Lines, planes, and hyperplanes Let w be a solution to Ax = b. Then

A(w − x0) = Aw − Ax0 = 0

so w − x0 ∈ N (A) and w ∈ x0 + N (A). Vector spaces

Paul Schrimpf

Normed vector spaces Examples Theorem Inner product Let A be an m by n matrix. Then dimN (A) = n − rankA spaces Useful inequalities

Projections Proof. Row, column, • N and null space Let u1, ..., uk be a basis for (A). We can add Row space Rn Column space uk+1, ..., un to u1, ..., uk to form a basis for . Null space • Show that Auk+1, ..., Aun are a basis for the column Applications Portfolio analysis space First and second • welfare theorems linearly independent Lines, planes, and • hyperplanes span ColA. Vector spaces

Paul Schrimpf

Normed vector spaces Theorem (Rouche-Capelli)´ Examples Inner product A system of linear equations with n variables has a solution spaces Useful inequalities if and only if the rank of its coefficient matrix, A, is equal to Projections the rank of its augmented matrix, A.ˆ Equivalently, a solution Row, column, exists if and only if b ∈ Col(A). and null space Row space If a solution exists and rankA is equal to its number of Column space Null space columns, the solution is unique. If a solution exists and Applications rankA is less than its number of columns, there are infinite Portfolio analysis First and second welfare theorems solutions. In this case the set of solutions forms is Lines, planes, and N hyperplanes x0 + (A), where x0 is any particular solution to Ax = b. This set of solutions is an affine subspace of dimension n − rankA. Vector spaces Paul Schrimpf Relationship among row,

Normed column, and null spaces vector spaces Examples

Inner product spaces Useful inequalities

Projections

Row, column, and null space • T ⊆ Rm Row space Col(A) = Row(A ) Column space T n Null space • Row(A) = Col(A ) ⊆ R Applications • N ⊆ Rn N T ⊆ Rm Portfolio analysis (A) and (A ) First and second welfare theorems • ∈ N ∈ ⟨ ⟩ Lines, planes, and Let x (A), y Row(A), what is x, y ? hyperplanes Vector spaces Paul Schrimpf Relationship among row,

Normed . column, and null spaces vector spaces Examples • If x ∈ N (A), y ∈ Row(A) = Col(AT ), then ⟨x, y⟩ = 0. Inner product n spaces • N (A) and Row(A) are orthogonal subspaces of R Useful inequalities • If x ∈ N (AT ), y ∈ Row(AT ) = Col(A), then ⟨x, y⟩ = 0. Projections • N (AT ) and Col(A) are orthogonal subspaces of Rm Row, column, • n and null space ∀x ∈ R , Row space Column space Null space Ax =A(PRowAx + PN (A)x) Applications Portfolio analysis =A(PRowAx) + A(PN (A)x) First and second T welfare theorems =A(P )x ∈ Col(A) = Row(A ) Lines, planes, and RowA hyperplanes • ∀w ∈ Rm,

T T A w =A (PColAw + PN (AT )w) T T =A (PColAw) + A (PN (AT )w) T T =A (PColA)w ∈ Row(A) = Col(A )