A Brief Review of Vector Spaces

In order to understand the geometry of OLS, we must have a firm grasp of concepts in the linear algebra of vector spaces . Specifically, we will be concerned with inner products, orthogonality, span, basis, and dimension . We start by defining a vector space.

Definition: A vector space is a set of vectors that is closed under scalar multiplication and addition.

Consider the 2-dimensional Euclidean vector space E 2 . Any 2-dimensional vector can

2 1 0 be written as a point in E . Consider the two vectors e1 =   and e2 =   . Any point 0 1

2 in E can be written as a linear combination of e1 and e2. For example, consider the

point (1,4). It can be written as: e1 + 4e2 .

A Euclidean space is endowed with an inner product . Consider two vectors, z1 and z2. The inner product of z1 and z2 is:

z1 ,z2 = z1 ' z2 = ∑ z1i z2i .

The length , or norm , of a vector, is computed as the square root of the inner product of a vector with itself:

z ≡ z, z .

The concept of orthogonality is intimately related to the inner product. Two vectors are said to be orthogonal if their inner product is zero. For example, consider the following

1 coordinates: (1,1) and (-2,2). These can be written as e1 + e2 and − 2e1 + 2e2 , or   and 1

− 2 − 2   . Orthogonality of these vectors is obvious: []1 1   = 0 .  2   2  We now introduce the concept of span . Span is concerned with all the possible linear combinations of a set of vectors. Consider the two vectors e1 and e2 discussed earlier. If we are interested in the span of e1 and e2, we wish to know what vector space is

2 comprised of all linear combinations of e1 and e2. Since any point in E can be written

2 as a linear combination of e1 and e2, it is said that the span of e1 and e2 is E , written as

2 S(e1 ,e2 ) = E .

γ  Consider the vector e1 in isolation. All linear combinations of e1 take the form   , 0

1 thus S(e1 ) = E = x . We can also consider the subspace orthogonal to that spanned by

0 ⊥ 1 e1. These vectors take the form   . We can thus write S (e1 ) = S(e2 ) = E = y . δ 

We now turn our attention to the concepts of basis and dimension . Note that the vectors e1 and e2 indeed span the 2-dimensional Euclidean space. However, if we

2 γ  2 included another vector in E , say   , then the set of three vectors also spans E . In δ  some sense, one vector is redundant. This brings us to the concept of basis :

Definition: A basis for a vector space is a set of linearly independent vectors that spans the vector space.

Definition: The dimension of a vector space is number of basis vectors for the space.