Chapter 3 Direct Sums, Affine Maps, the Dual Space, Duality

Chapter 3

Direct Sums, Ane Maps, The Dual Space, Duality

3.1 Direct Products, Sums, and Direct Sums

There are some useful ways of forming new vector spaces from older ones. Deﬁnition 3.1. Given p 2vectorspacesE ,...,E , 1 p the product F = E1 Ep can be made into a vector space by deﬁning addition⇥···⇥ and scalar multiplication as follows:

(u1,...,up)+(v1,...,vp)=(u1 + v1,...,up + vp) (u1,...,up)=(u1,...,up), for all ui,vi Ei and all R. 2 2 With the above addition and multiplication, the vector space F = E E is called the direct product of 1 ⇥···⇥ p the vector spaces E1,...,Ep.

151 152 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY The projection maps pr : E E E given by i 1 ⇥···⇥ p ! i pri(u1,...,up)=ui are clearly linear.

Similarly, the maps in : E E E given by i i ! 1 ⇥···⇥ p ini(ui)=(0,...,0,ui, 0,...,0) are injective and linear.

It can be shown (using bases) that dim(E E )=dim(E )+ +dim(E ). 1 ⇥···⇥ p 1 ··· p

Let us now consider a vector space E and p subspaces U1,...,Up of E.

We have a map a: U U E 1 ⇥···⇥ p ! given by a(u ,...,u )=u + + u , 1 p 1 ··· p with u U for i =1,...,p. i 2 i 3.1. DIRECT PRODUCTS, SUMS, AND DIRECT SUMS 153 It is clear that this map is linear, and so its image is a subspace of E denoted by U + + U 1 ··· p and called the sum of the subspaces U1,...,Up.

By deﬁnition, U + + U = u + + u u U , 1 i p , 1 ··· p { 1 ··· p | i 2 i   } and it is immediately veriﬁed that U + + U is the 1 ··· p smallest subspace of E containing U1,...,Up.

If the map a is injective, then Ker a = (0,...,0 ) , { } p which means that if ui Ui for i =1,...,p and if 2 | {z } u + + u =0 1 ··· p then u1 =0,...,up =0.

In this case, every u U1 + + Up has a unique expression as a sum 2 ··· u = u + + u , 1 ··· p with u U ,fori =1,...,p. i 2 i 154 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY It is also clear that for any p nonzero vectors u U , i 2 i u1,...,up are linearly independent.

Deﬁnition 3.2. For any vector space E and any p 2 subspaces U1,...,Up of E,ifthemapa deﬁned above is injective, then the sum U1 + + Up is called a direct sum and it is denoted by ··· U U . 1 ··· p The space E is the direct sum of the subspaces Ui if E = U U . 1 ··· p

Observe that when the map a is injective, then it is a linear isomorphism between U1 Up and U U . ⇥···⇥ 1 ··· p The di↵erence is that U U is deﬁned even if 1 ⇥···⇥ p the spaces Ui are not assumed to be subspaces of some common space.

There are natural injections from each Ui to E denoted by in : U E. i i ! 3.1. DIRECT PRODUCTS, SUMS, AND DIRECT SUMS 155 Now, if p =2,itiseasytodeterminethekernelofthe map a: U U E.Wehave 1 ⇥ 2 ! a(u ,u )=u + u =0i↵u = u ,u U ,u U , 1 2 1 2 1 2 1 2 1 2 2 2 which implies that Ker a = (u, u) u U U . { | 2 1 \ 2}

Now, U U is a subspace of E and the linear map 1 \ 2 u (u, u)isclearlyanisomorphismbetweenU1 U2 and7! Ker a,soKera is isomorphic to U U . \ 1 \ 2 As a consequence, we get the following result:

Proposition 3.1. Given any vector space E and any two subspaces U1 and U2, the sum U1 + U2 is a direct sum i↵ U U =(0). 1 \ 2 156 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY Recall that an n n matrix A M is symmetric if ⇥ 2 n A> = A, skew -symmetric if A> = A.Itisclearthat

S(n)= A M A> = A { 2 n | } Skew(n)= A M A> = A { 2 n | } are subspaces of M ,andthatS(n) Skew(n)=(0). n \ Observe that for any matrix A M ,thematrixH(A)= 2 n (A + A>)/2issymmetricandthematrix S(A)=(A A>)/2isskew-symmetric.Since

A + A> A A> A = H(A)+S(A)= + , 2 2 we have the direct sum

M = S(n) Skew(n). n 3.1. DIRECT PRODUCTS, SUMS, AND DIRECT SUMS 157 Proposition 3.2. Given any vector space E and any p 2 subspaces U1,...,Up, the following properties are equivalent: (1) The sum U + + U is a direct sum. 1 ··· p (2) We have p U U =(0),i=1,...,p. i \ j ✓ j=1,j=i ◆ X6 (3) We have i 1 U U =(0),i=2,...,p. i \ j j=1 ✓ X ◆

The isomorphism U U U U implies 1 ⇥···⇥ p ⇡ 1 ··· p

Proposition 3.3. If E is any vector space, for any (ﬁnite-dimensional) subspaces U1,..., Up of E, we have dim(U U )=dim(U )+ +dim(U ). 1 ··· p 1 ··· p 158 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY If E is a direct sum E = U U , 1 ··· p since every u E can be written in a unique way as 2 u = u + + u 1 ··· p for some ui Ui for i =1...,p,wecandeﬁnethemaps ⇡ : E U ,called2 projections,by i ! i ⇡ (u)=⇡ (u + + u )=u . i i 1 ··· p i

It is easy to check that these maps are linear and satisfy the following properties:

⇡i if i = j ⇡j ⇡i = 0ifi = j, ( 6 ⇡ + + ⇡ =id . 1 ··· p E 3.1. DIRECT PRODUCTS, SUMS, AND DIRECT SUMS 159 For example, in the case of the direct sum M = S(n) Skew(n), n the projection onto S(n)isgivenby

A + A ⇡ (A)=H(A)= >, 1 2 and the projection onto Skew(n)isgivenby

A A> ⇡ (A)=S(A)= . 2 2

Clearly, H(A)+S(A)=A, H(H(A)) = H(A), S(S(A)) = S(A), and H(S(A)) = S(H(A)) = 0.

Afunctionf such that f f = f is said to be idempotent. Thus, the projections ⇡i are idempotent.

Conversely, the following proposition can be shown: 160 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY Proposition 3.4. Let E be a vector space. For any p 2 linear maps f : E E, if i ! fi if i = j fj fi = 0 if i = j, ( 6 f + + f =id , 1 ··· p E then if we let Ui = fi(E), we have a direct sum E = U U . 1 ··· p

We also have the following proposition characterizing idempotent linear maps whose proof is also left as an exercise.

Proposition 3.5. For every vector space E, if f : E E is an idempotent linear map, i.e., f f = f, then we! have a direct sum E =Kerf Im f, so that f is the projection onto its image Im f.

We are now ready to prove a very crucial result relating the rank and the dimension of the kernel of a linear map. 3.1. DIRECT PRODUCTS, SUMS, AND DIRECT SUMS 161 Theorem 3.6. Let f : E F be a linear map. For ! any choice of a basis (f1,...,fr) of Im f, let (u1,...,ur) be any vectors in E such that fi = f(ui), for i = 1,...,r.Ifs:Imf E is the unique linear map de- ! ﬁned by s(fi)=ui, for i =1,...,r, then s is injective, f s =id, and we have a direct sum E =Kerf Im s as illustrated by the following diagram: f / Ker f / E =Kerf Im s Im f F. o s ✓ As a consequence, dim(E)=dim(Kerf)+dim(Imf)=dim(Kerf)+rk(f).

Remark: The dimension dim(Ker f)ofthekernelofa linear map f is often called the nullity of f.

We now derive some important results using Theorem 3.6. 162 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY Proposition 3.7. Given a vector space E, if U and V are any two subspaces of E, then dim(U)+dim(V )=dim(U + V )+dim(U V ), \ an equation known as Grassmann’s relation.

The Grassmann relation can be very useful to ﬁgure out whether two subspace have a nontrivial intersection in spaces of dimension > 3.

For example, it is easy to see that in R5,therearesub- spaces U and V with dim(U)=3anddim(V )=2such that U V =(0). \ However, we can show that if dim(U)=3anddim(V )= 3, then dim(U V ) 1. \ As another consequence of Proposition 3.7, if U and V are two hyperplanes in a vector space of dimension n,so that dim(U)=n 1anddim(V )=n 1, we have dim(U V ) n 2, \ and so, if U = V ,then 6 dim(U V )=n 2. \ 3.1. DIRECT PRODUCTS, SUMS, AND DIRECT SUMS 163

Proposition 3.8. If U1,...,Up are any subspaces of a ﬁnite dimensional vector space E, then dim(U + + U ) dim(U )+ +dim(U ), 1 ··· p  1 ··· p and dim(U + + U )=dim(U )+ +dim(U ) 1 ··· p 1 ··· p i↵the U s form a direct sum U U . i 1 ··· p

Another important corollary of Theorem 3.6 is the following result:

Proposition 3.9. Let E and F be two vector spaces with the same ﬁnite dimension dim(E)=dim(F )= n. For every linear map f : E F , the following properties are equivalent: ! (a) f is bijective. (b) f is surjective. (c) f is injective. (d) Ker f =(0). 164 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY One should be warned that Proposition 3.9 fails in inﬁnite dimension.

We also have the following basic proposition about injective or surjective linear maps.

Proposition 3.10. Let E and F be vector spaces, and let f : E F be a linear map. If f : E F is injective, then! there is a surjective linear map!r : F ! E called a retraction, such that r f =idE.Iff : E F is surjective, then there is an injective linear map! s: F E called a section, such that f s =id . ! F

The notion of rank of a linear map or of a matrix important, both theoretically and practically, since it is the key to the solvability of linear equations.

Proposition 3.11. Given a linear map f : E F , the following properties hold: ! (i) rk(f)+dim(Kerf)=dim(E). (ii) rk(f) min(dim(E), dim(F )).  3.1. DIRECT PRODUCTS, SUMS, AND DIRECT SUMS 165 The rank of a matrix is deﬁned as follows.

Deﬁnition 3.3. Given a m n-matrix A =(aij), the rank rk(A)ofthematrixA is⇥ the maximum number of linearly independent columns of A (viewed as vectors in Rm).

In view of Proposition 1.4, the rank of a matrix A is the dimension of the subspace of Rm generated by the columns of A.

Let E and F be two vector spaces, and let (u1,...,un)be abasisofE,and(v1,...,vm)abasisofF .Letf : E F be a linear map, and let M(f)beitsmatrixw.r.t.the! bases (u1,...,un)and(v1,...,vm). 166 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY Since the rank rk(f)off is the dimension of Im f,which is generated by (f(u1),...,f(un)), the rank of f is the maximum number of linearly independent vectors in (f(u1),...,f(un)), which is equal to the number of linearly independent columns of M(f), since F and Rm are isomorphic.

Thus, we have rk(f)=rk(M(f)), for every matrix representing f.

We will see later, using duality, that the rank of a matrix A is also equal to the maximal number of linearly independent rows of A. 3.1. DIRECT PRODUCTS, SUMS, AND DIRECT SUMS 167

Figure 3.1: How did Newton start a business 168 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY 3.2 Ane Maps

We showed in Section 1.5 that every linear map f must send the zero vector to the zero vector, that is,

f(0) = 0.

Yet, for any ﬁxed nonzero vector u E (where E is any 2 vector space), the function tu given by

t (x)=x + u, for all x E u 2 shows up in pratice (for example, in robotics).

Functions of this type are called translations.Theyare not linear for u =0,sincet (0) = 0 + u = u. 6 u More generally, functions combining linear maps and translations occur naturally in many applications (robotics, computer vision, etc.), so it is necessary to understand some basic properties of these functions. 3.2. AFFINE MAPS 169 For this, the notion of ane combination turns out to play a key role.

Recall from Section 1.5 that for any vector space E,given any family (ui)i I of vectors ui E,anane combina- 2 2 tion of the family (ui)i I is an expression of the form 2

iui with i =1, i I i I X2 X2 where (i)i I is a family of scalars. 2 Alinearcombinationplacesnorestrictiononthescalars involved, but an ane combination is a linear combination, with the restriction that the scalars i must add up to 1.Nevertheless,alinearcombinationcanalways be viewed as an ane combination, using 0 with the co- ecient 1 i I i. 2 Ane combinationsP are also called barycentric combinations.

Although this is not obvious at ﬁrst glance, the condition that the scalars i add up to 1 ensures that ane combinations are preserved under translations. 170 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY To make this precise, consider functions f : E F , where E and F are two vector spaces, such that! there is some linear map h: E F and some ﬁxed vector b F (a translation vector!), such that 2 f(x)=h(x)+b, for all x E. 2 The map f given by

x 8/5 6/5 x 1 1 1 + x2 7! 3/10 2/5 x2 1 ✓ ◆ ✓ ◆✓ ◆ ✓ ◆ is an example of the composition of a linear map with a translation.

We claim that functions of this type preserve ane combinations. 3.2. AFFINE MAPS 171 Proposition 3.12. For any two vector spaces E and F , given any function f : E F deﬁned such that ! f(x)=h(x)+b, for all x E, 2 where h: E F is a linear map and b is some ﬁxed ! vector in F , for every ane combination i I iui 2 (with i I i =1), we have 2 P P f iui = if(ui). ✓ i I ◆ i I X2 X2 In other words, f preserves ane combinations.

Surprisingly, the converse of Proposition 3.12 also holds. 172 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY Proposition 3.13. For any two vector spaces E and F , let f : E F be any function that preserves ane combinations,! i.e., for every ane combination

i I iui (with i I i =1), we have 2 2 P P f iui = if(ui). ✓ i I ◆ i I X2 X2 Then, for any a E, the function h: E F given by 2 !

h(x)=f(a + x) f(a) is a linear map independent of a, and

f(a + x)=f(a)+h(x), for all x E. 2

In particular, for a =0, if we let c = f(0), then

f(x)=c + h(x), for all x E. 2 3.2. AFFINE MAPS 173 We should think of a as a chosen origin in E.

The function f maps the origin a in E to the origin f(a) in F .

Proposition 3.13 shows that the deﬁnition of h does not depend on the origin chosen in E.Also,since

f(x)=c + h(x), for all x E 2 for some ﬁxed vector c F ,weseethatf is the com- 2 position of the linear map h with the translation tc (in F ).

The unique linear map h as above is called the linear map associated with f and it is sometimes denoted by !f .

Observe that the linear map associated with a pure translation is the identity.

In view of Propositions 3.12 and 3.13, it is natural to make the following definition. 174 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY Definition 3.4. For any two vector spaces E and F ,a function f : E F is an ane map if f preserves ane ! combinations, i.e.,foreveryanecombination i I iui 2 (with i I i =1),wehave 2 P P f iui = if(ui). ✓ i I ◆ i I X2 X2 Equivalently, a function f : E F is an ane map if ! there is some linear map h: E F (also denoted by !f ) and some fixed vector c F such! that 2 f(x)=c + h(x), for all x E. 2

Note that a linear map always maps the standard origin 0inE to the standard origin 0 in F .

However an ane map usually maps 0 to a nonzero vector c = f(0). This is the “translation component” of the ane map. 3.2. AFFINE MAPS 175 When we deal with ane maps, it is often fruitful to think of the elements of E and F not only as vectors but also as points.

In this point of view, points can only be combined using ane combinations,butvectorscanbecombinedinan unrestricted fashion using linear combinations.

We can also think of u + v as the result of translating the point u by the translation tv.

These ideas lead to the deﬁnition of ane spaces,but this would lead us to far aﬁeld, and for our purposes, it is enough to stick to vector spaces.

Still, one should be aware that ane combinations really apply to points, and that points are not vectors!

If E and F are ﬁnite dimensional vector spaces, with dim(E)=n and dim(F )=m,thenitisusefultorepre- sent an ane map with respect to bases in E in F . 176 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY However, the translation part c of the ane map must be somewhow incorporated.

There is a standard trick to do this which amounts to viewing an ane map as a linear map between spaces of dimension n +1andm +1.

We also have the extra ﬂexibility of choosing origins, a E and b F . 2 2

Let (u1,...,un)beabasisofE,(v1,...,vm)beabasis of F ,andleta E and b F be any two ﬁxed vectors viewed as origins2 . 2

Our ane map f has the property that if v = f(u), then

v b = f(a + u a) b = f(a) b + h(u a).

So, if we let y = v b, x = u a,andd = f(a) b,then y = h(x)+d, x E. 2 3.2. AFFINE MAPS 177 Over the basis =(u ,...,u ), we write U 1 n x = x u + + x u , 1 1 ··· n n and over the basis =(v ,...,v ), we write V 1 m

y = y v + + y v , 1 1 ··· m m d = d v + + d v . 1 1 ··· m m Then, since y = h(x)+d, if we let A be the m n matrix representing the linear map h,thatis,thejth⇥ column of A consists of the coordinates of h(uj)overthebasis(v1,...,vm), then we can write

y = Ax + d . V U V where x =(x1,...,xn)>, y =(y1,...,ym)>,and U V d =(d1,...,dm)>. V This is the matrix representation of our ane map f. 178 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY The reason for using the origins a and b is that it gives us more ﬂexibility.

In particular, when E = F ,ifthereissomea E such that f(a)=a (a is a ﬁxed point of f), then we2 can pick b = a.

Then, because f(a)=a,weget v = f(u)=f(a+u a)=f(a)+h(u a)=a+h(u a), that is v a = h(u a). With respect to the new origin a,ifwedeﬁnex and y by x = u a y = v a, then we get y = h(x).

Then, f really behaves like a linear map, but with respect to the new origin a (not the standard origin 0). This is the case of a rotation around an axis that does not pass through the origin. 3.2. AFFINE MAPS 179

Remark: Apair(a, (u1,...,un)) where (u1,...,un)is abasisofE and a is an origin chosen in E is called an ane frame.

We now describe the trick which allows us to incorporate the translation part d into the matrix A.

We deﬁne the (m+1) (n+1) matrix A0 obtained by ﬁrst adding d as the (n +1)thcolumn,andthen(0⇥ ,...,0, 1) n as the (m +1)throw: | {z } Ad A = . 0 0 1 ✓ n ◆ Then, it is clear that

y Ad x = 1 0 1 1 ✓ ◆ ✓ n ◆✓ ◆ i↵

y = Ax + d. 180 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY This amounts to considering a point x Rn as a point 2 n+1 (x, 1) in the (ane) hyperplane Hn+1 in R of equation xn+1 =1.

Then, an ane map is the restriction to the hyperplane n+1 m+1 Hn+1 of the linear map f from R to R corre- sponding to the matrix A0,whichmapsHn+1 into Hm+1 (f(H ) H ). b n+1 ✓ m+1 Figureb 3.2 illustrates this process for n =2.

(x1,x2, 1)

H3 : x3 =1

x =(x1,x2)

Figure 3.2: Viewing Rn as a hyperplane in Rn+1 (n =2) 3.2. AFFINE MAPS 181 For example, the map

x 11 x 3 1 1 + x2 7! 13 x2 0 ✓ ◆ ✓ ◆✓ ◆ ✓ ◆ deﬁnes an ane map f which is represented in R3 by

x1 113 x1 x2 130 x2 . 0 1 1 7! 00011 0 1 1 @ A @ A @ A It is easy to check that the point a =(6, 3) is ﬁxed by f,whichmeansthatf(a)=a,sobytranslatingthe coordinate frame to the origin a,theanemapbehaves like a linear map.

The idea of considering Rn as an hyperplane in Rn+1 can be used to deﬁne projective maps. 182 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY

Al\ cal<) have +,ov( lejs. 1h'O..{e -ro\.l( '). I OJ¥' q cat

Figure 3.3: Dog Logic 3.3. THE DUAL SPACE E⇤ AND LINEAR FORMS 183

3.3 The Dual Space E⇤ and Linear Forms

We already observed that the ﬁeld K itself (K = R or K = C)isavectorspace(overitself).

The vector space Hom(E,K)oflinearmapsfromE to the ﬁeld K,thelinear forms,playsaparticularrole.

We take a quick look at the connection between E and E⇤ =Hom(E,K), its dual space.

As we will see shortly, every linear map f : E F gives ! rise to a linear map f > : F ⇤ E⇤,anditturnsoutthat ! in a suitable basis, the matrix of f > is the transpose of the matrix of f.

Thus, the notion of dual space provides a conceptual ex- planation of the phenomena associated with transposi- tion.

But it does more, because it allows us to view subspaces as solutions of sets of linear equations and vice-versa. 184 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY Consider the following set of two “linear equations” in R3,

x y + z =0 x y z =0, and let us find out what is their set V of common solutions (x, y, z) R3. 2 By subtracting the second equation from the first, we get 2z =0,andbyaddingthetwoequations,wefindthat 2(x y)=0,sothesetV of solutions is given by

y = x z =0.

This is a one dimensional subspace of R3.Geometrically, this is the line of equation y = x in the plane z =0. 3.3. THE DUAL SPACE E⇤ AND LINEAR FORMS 185 Now, why did we say that the above equations are linear?

This is because, as functions of (x, y, z), both maps f1 :(x, y, z) x y + z and f2 :(x, y, z) x y z are linear. 7! 7!

The set of all such linear functions from R3 to R is a vector space; we used this fact to form linear combinations of the “equations” f1 and f2.

Observe that the dimension of the subspace V is 1.

The ambient space has dimension n =3andthereare two “independent” equations f1,f2,soitappearsthat the dimension dim(V )ofthesubspaceV deﬁned by m independent equations is dim(V )=n m, which is indeed a general fact (proved in Theorem 3.14). 186 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY More generally, in Rn,alinearequationisdeterminedby n an n-tuple (a1,...,an) R ,andthesolutionsofthis linear equation are given2 by the n-tuples (x ,...,x ) 1 n 2 Rn such that

a x + + a x =0; 1 1 ··· n n these solutions constitute the kernel of the linear map (x ,...,x ) a x + + a x . 1 n 7! 1 1 ··· n n The above considerations assume that we are working in n the canonical basis (e1,...,en)ofR ,butwecandefine “linear equations” independently of bases and in any dimension, by viewing them as elements of the vector space Hom(E,K)oflinearmapsfromE to the field K. 3.3. THE DUAL SPACE E⇤ AND LINEAR FORMS 187 Definition 3.5. Given a vector space E,thevector space Hom(E,K)oflinearmapsfromE to K is called the dual space (or dual) of E.ThespaceHom(E,K)is also denoted by E⇤,andthelinearmapsinE⇤ are called the linear forms,orcovectors.ThedualspaceE⇤⇤ of the space E⇤ is called the bidual of E.

As a matter of notation, linear forms f : E K will also ! be denoted by starred symbol, such as u⇤, x⇤,etc.

If E is a vector space of ﬁnite dimension n and (u1,...,un) is a basis of E,foranylinearformf ⇤ E⇤,forevery x = x u + + x u E,bylinearitywehave2 1 1 ··· n n 2

f ⇤(x)=f ⇤(u )x + + f ⇤(n )x 1 1 ··· n n = x + + x , 1 1 ··· n n with = f ⇤(u ) K for every i,1 i n. i i 2   188 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY

Thus, with respect to the basis (u1,...,un), the linear form f ⇤ is represented by the row vector ( ), 1 ··· n we have

x1 . f ⇤(x)= 1 n . , ··· 0x 1 n @ A alinearcombinationofthecoordinatesofx,andwecan view the linear form f ⇤ as a linear equation.

If we decide to use a column vector of coecients

c1 c = . 0 1 cn instead of a row vector, then@ theA linear form f ⇤ is deﬁned by

f ⇤(x)=c>x.

The above notation is often used in machine learning. 3.3. THE DUAL SPACE E⇤ AND LINEAR FORMS 189 Example 3.1. Given any di↵erentiable function f : Rn R,bydeﬁnition,foranyx Rn,thetotal ! 2 n derivative df x of f at x is the linear form df x : R R n ! deﬁned so that for all u =(u1,...,un) R , 2

u1 n @f @f . @f df x(u)= (x) (x) . = (x) ui. @x ··· @x 0 1 @x 1 n u i=1 i ✓ ◆ n X @ A

Example 3.2. Let ([0, 1]) be the vector space of contin- C uous functions f :[0, 1] R.Themap : ([0, 1]) R given by ! I C !

1 (f)= f(x)dx for any f ([0, 1]) I 2C Z0 is a linear form (integration). 190 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY

Example 3.3. Consider the vector space Mn(R)ofreal n n matrices. Let tr: Mn(R) R be the function given⇥ by !

tr(A)=a + a + + a , 11 22 ··· nn called the trace of A.Itisalinearform.

Let s:Mn(R) R be the function given by ! n

s(A)= aij, i,j=1 X where A =(aij). It is immediately veriﬁed that s is a linear form. 3.3. THE DUAL SPACE E⇤ AND LINEAR FORMS 191

Given a linear form u⇤ E⇤ and a vector v E,the 2 2 result u⇤(v)ofapplyingu⇤ to v is also denoted by u⇤,v . h i

This deﬁnes a binary operation , : E⇤ E K satisfying the following properties:h i ⇥ !

u⇤ + u⇤,v = u⇤,v + u⇤,v h 1 2 i h 1 i h 2 i u⇤,v + v = u⇤,v + u⇤,v h 1 2i h 1i h 2i u⇤,v = u⇤,v h i h i u⇤,v = u⇤,v . h i h i

The above identities mean that , is a bilinear map, since it is linear in each argument.h i

It is often called the canonical pairing between E⇤ and E. 192 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY In view of the above identities, given any ﬁxed vector v 2 E,themapevalv : E⇤ K (evaluation at v)deﬁned such that !

eval (u⇤)= u⇤,v = u⇤(v)foreveryu⇤ E⇤ v h i 2 is a linear map from E⇤ to K,thatis,evalv is a linear form in E⇤⇤.

Again from the above identities, the map eval : E E⇤⇤,deﬁnedsuchthat E ! eval (v)=eval for every v E, E v 2 is a linear map.

We shall see that it is injective, and that it is an isomorphism when E has ﬁnite dimension. 3.3. THE DUAL SPACE E⇤ AND LINEAR FORMS 193 We now formalize the notion of the set V 0 of linear equations vanishing on all vectors in a given subspace V E, and the notion of the set U 0 of common solutions✓ of a given set U E⇤ of linear equations. ✓ The duality theorem (Theorem 3.14) shows that the dimensions of V and V 0,andthedimensionsofU and U 0, are related in a crucial way.

It also shows that, in ﬁnite dimension, the maps V V 0 and U U 0 are inverse bijections from subspaces7! of E 7! to subspaces of E⇤. 194 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY

Deﬁnition 3.6. Given a vector space E and its dual E⇤, we say that a vector v E and a linear form u⇤ E⇤ 2 2 are orthogonal i↵ u⇤,v =0.GivenasubspaceV of h i E and a subspace U of E⇤,wesaythatV and U are orthogonal i↵ u⇤,v =0foreveryu⇤ U and every h i 2 v V .GivenasubsetV of E (resp. a subset U of E⇤), 2 0 0 the orthogonal V of V is the subspace V of E⇤ deﬁned such that

0 V = u⇤ E⇤ u⇤,v =0, for every v V { 2 |h i 2 }

(resp. the orthogonal U 0 of U is the subspace U 0 of E deﬁned such that

0 U = v E u⇤,v =0, for every u⇤ U ). { 2 |h i 2 }

0 The subspace V E⇤ is also called the annihilator of V . ✓ 3.3. THE DUAL SPACE E⇤ AND LINEAR FORMS 195

0 The subspace U E annihilated by U E⇤ does not have a special name.✓ It seems reasonable✓ to call it the linear subspace (or linear variety) deﬁned by U.

Informally, V 0 is the set of linear equations that vanish on V ,andU 0 is the set of common zeros of all linear equations in U.WecanalsodeﬁneV 0 by

0 V = u⇤ E⇤ V Ker u⇤ { 2 | ✓ } and U 0 by

0 U = Ker u⇤. u U \⇤2

0 0 Observe that E = 0 =(0),and 0 = E⇤. { } { } 0 0 Furthermore, if V1 V2 E,thenV2 V1 E⇤,and ✓ ✓0 0 ✓ ✓ if U U E⇤,thenU U E. 1 ✓ 2 ✓ 2 ✓ 1 ✓ 196 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY It can also be shown that that V V 00 for every subspace V of E,andthatU U 00 for✓ every subspace U of ✓ E⇤.

We will see shortly that in ﬁnite dimension, we have

V = V 00 and U = U 00.

Here are some examples. Let E =M2(R), the space of real 2 2matrices,andletV be the subspace of M2(R) spanned⇥ by the matrices

01 10 00 , , . 10 00 01 ✓ ◆ ✓ ◆ ✓ ◆ We check immediately that the subspace V consists of all matrices of the form

ba , ac ✓ ◆ that is, all symmetric matrices. 3.3. THE DUAL SPACE E⇤ AND LINEAR FORMS 197 The matrices

a11 a12 a a ✓ 21 22◆ in V satisfy the equation

a a =0, 12 21 and all scalar multiples of these equations, so V 0 is the subspace of E⇤ spanned by the linear form given by

u⇤(a ,a ,a ,a )=a a . 11 12 21 22 12 21 By the duality theorem (Theorem 3.14) we have

dim(V 0)=dim(E) dim(V )=4 3=1.

The above example generalizes to E =Mn(R)forany n 1, but this time, consider the space U of linear forms asserting that a matrix A is symmetric; these are the linear forms spanned by the n(n 1)/2equations a a =0, 1 i

a a =0, 1 i = j n ij ji  6  are redundant. It is easy to check that the equations (linear forms) for which i

To be more precise, let U be the space of linear forms in E⇤ spanned by the linear forms

uij⇤ (a11,...,a1n,a21,...,a2n,...,an1,...,ann) = a a , 1 i

The dimension of U is n(n 1)/2. Then, the set U 0 of common solutions of these equations is the space S(n)of symmetric matrices.

By the duality theorem (Theorem 3.14), this space has dimension

n(n +1) n(n 1) = n2 . 2 2 3.3. THE DUAL SPACE E⇤ AND LINEAR FORMS 199

If E =Mn(R), consider the subspace U of linear forms in E⇤ spanned by the linear forms

uij⇤ (a11,...,a1n,a21,...,a2n,...,an1,...,ann) = a + a , 1 i

It is easy to see that these linear forms are linearly independent, so dim(U)=n(n +1)/2.

0 The space U of matrices A Mn(R)satifyingallofthe above equations is clearly the2 space Skew(n)ofskew- symmetric matrices.

By the duality theorem (Theorem 3.14), the dimension of U 0 is

n(n 1) n(n +1) = n2 . 2 2 200 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY

For yet another example with E =Mn(R), for any A 2 Mn(R), consider the linear form in E⇤ given by

tr(A)=a + a + + a , 11 22 ··· nn called the trace of A.

The subspace U 0 of E consisting of all matrices A such that tr(A)=0isaspaceofdimensionn2 1. The dimension equations dim(V )+dim(V 0)=dim(E) dim(U)+dim(U 0)=dim(E) are always true (if E is ﬁnite-dimensional). This is part of the duality theorem (Theorem 3.14).

In constrast with the previous examples, given a matrix A Mn(R), the equations asserting that A>A = I are not2 linear constraints. 3.3. THE DUAL SPACE E⇤ AND LINEAR FORMS 201 For example, for n =2,wehave 2 2 a11 + a21 =1 2 2 a21 + a22 =1 a11a12 + a21a22 =0.

Given a vector space E and any basis (ui)i I for E,we 2 can associate to each u alinearformu⇤ E⇤,andthe i i 2 ui⇤ have some remarkable properties.

Deﬁnition 3.7. Given a vector space E and any basis (ui)i I for E,byProposition1.10,foreveryi I,there 2 2 is a unique linear form ui⇤ such that 1ifi = j u (u )= i⇤ j 0ifi = j, ⇢ 6 for every j I.Thelinearformu⇤ is called the coordi- 2 i nate form of index i w.r.t. the basis (ui)i I. 2 202 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY Remark: Given an index set I,authorsoftendeﬁnethe so called Kronecker symbol ij,suchthat 1ifi = j = ij 0ifi = j, ⇢ 6 for all i, j I. 2 Then, ui⇤(uj)=ij.

The reason for the terminology coordinate form is as follows: If E has ﬁnite dimension and if (u1,...,un)isa basis of E,foranyvector v = u + + u , 1 1 ··· n n we have

ui⇤(v)=i.

Therefore, ui⇤ is the linear function that returns the ith coordinate of a vector expressed over the basis (u1,...,un).

As part of the Duality theorem (Theorem 3.14), we will prove that if (u1,...,un)isabasisofE,then(u1⇤,...,un⇤) is a basis of E⇤ called the dual basis. 3.3. THE DUAL SPACE E⇤ AND LINEAR FORMS 203 n n If (u1,...,un)isabasisofR (more generally K ), it is possible to ﬁnd explicitly the dual basis (u1⇤,...,un⇤), where each ui⇤ is represented by a row vector.

For example, consider the columns of the B´ezier matrix

1 33 1 03 63 B = . 4 000 3 31 B00 0 1C B C @ A

The form u1⇤ is represented by a row vector (1 2 3 4) such that

1 33 1 03 63 = 1000 . 1 2 3 4 000 3 31 B00 0 1C B C @ A

This implies that u1⇤ is the ﬁrst row of the inverse of B4. 204 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY Since

11 11 01/32/31 B 1 = , 4 0001/311 B00 01C B C @ A the linear forms (u1⇤,u2⇤,u3⇤,u4⇤)correspondtotherowsof 1 B4 .

In particular, u1⇤ is represented by (1 1 1 1).

The above method works for any n.Givenanybasis n (u1,...,un)ofR ,ifP is the n n matrix whose jth ⇥ column is uj,thenthedualformui⇤ is given by the ith 1 row of the matrix P .

We have the following important duality theorem adapted from E. Artin. 3.3. THE DUAL SPACE E⇤ AND LINEAR FORMS 205 Theorem 3.14. (Duality theorem) Let E be a vector space of dimension n. The following properties hold:

(a) For every basis (u1,...,un) of E, the family of coordinate forms (u1⇤,...,un⇤) is a basis of E⇤. (b) For every subspace V of E, we have V 00 = V . (c) For every pair of subspaces V and W of E such that E = V W , with V of dimension m, for every basis (u1,...,un) of E such that (u1,...,um) is a basis of V and (um+1,...,un) is a basis of W , the 0 family (u1⇤,...,um⇤ ) is a basis of the orthogonal W 00 of W in E⇤. Furthermore, we have W = W , and dim(W )+dim(W 0)=dim(E).

(d) For every subspace U of E⇤, we have dim(U)+dim(U 0)=dim(E), where U 0 is the orthogonal of U in E, and U 00 = U. 206 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY Part (a) of Theorem 3.14 shows that

dim(E)=dim(E⇤), and if (u1,...,un)isabasisofE,then(u1⇤,...,un⇤)is abasisofthedualspaceE⇤ called the dual basis of (u1,...,un).

Deﬁne the function from subspaces of E to subspaces of E E⇤ and the function from subspaces of E⇤ to subspaces of E by Z

(V )=V 0,V E E 0 ✓ (U)=U ,U E⇤. Z ✓

By part (c) and (d) of theorem 3.14,

( )(V )=V 00 = V ZE ( )(U)=U 00 = U, EZ so =idand =id,andthemaps and are inverseZE bijections. EZ E V 3.3. THE DUAL SPACE E⇤ AND LINEAR FORMS 207 These maps set up a duality between subspaces of E,and subspaces of E⇤.

One should be careful that this bijection does not hold if E has inﬁnite dimension. Some restrictions on the dimensions of U and V are needed.

Suppose that V is a subspace of Rn of dimension m and that (v1,...,vm)isabasisofV .

0 To ﬁnd a basis of V ,weﬁrstextend(v1,...,vm)toa n basis (v1,...,vn)ofR ,andthenbypart(c)ofTheorem 0 3.14, we know that (vm⇤ +1,...,vn⇤)isabasisofV .

For example, suppose that V is the subspace of R4 spanned by the two linearly independent vectors

1 1 1 1 v = v = , 1 011 2 0 11 B1C B 1C B C B C @ A @ A the ﬁrst two vectors of the Haar basis in R4. 208 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY The four columns of the Haar matrix

11 1 0 11 10 W = 01 10 11 B1 10 1C B C @ A form a basis of R4,andtheinverseofW is given by 1/41/41/41/4 1/41/4 1/4 1/4 W 1 = . 01/2 1/20 01 B 001/2 1/2C B C @ A

Since the dual basis (v1⇤,v2⇤,v3⇤,v4⇤)isgivenbytherowof 1 1 W ,thelasttworowsofW ,

1/2 1/20 0 , 001 /2 1/2 ✓ ◆ form a basis of V 0. 3.3. THE DUAL SPACE E⇤ AND LINEAR FORMS 209 We also obtain a basis by rescaling by the factor 1/2, so the linear forms given by the row vectors

1 10 0 001 1 ✓ ◆ form a basis of V 0,thespaceoflinearforms(linearequa- tions) that vanish on the subspace V .

The method that we described to ﬁnd V 0 requires ﬁrst extending a basis of V and then inverting a matrix, but there is a more direct method. 210 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY Indeed, let A be the n m matrix whose columns are ⇥ the basis vectors (v1,...,vm)ofV .Then,alinearform 0 u represented by a row vector belongs to V i↵ uvi =0 for i =1,...,m i↵

uA =0 i↵

A>u> =0.

Therefore, all we need to do is to ﬁnd a basis of the nullspace of A>.

This can be done quite e↵ectively using the reduction of amatrixtoreducedrowechelonform(rref);seeSection 4.5. 3.3. THE DUAL SPACE E⇤ AND LINEAR FORMS 211 Here is another example illustrating the power of Theo- rem 3.14.

Let E =Mn(R), and consider the equations asserting that the sum of the entries in every row of a matrix 2 Mn(R)isequaltothesamenumber.

We have n 1equations n (a a )=0, 1 i n 1, ij i+1j   j=1 X and it is easy to see that they are linearly independent.

Therefore, the space U of linear forms in E⇤ spanned by the above linear forms (equations) has dimension n 1, and the space U 0 of matrices sastisfying all these equations has dimension n2 n +1. It is not so obvious to ﬁnd a basis for this space. 212 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY

When E is of ﬁnite dimension n and (u1,...,un)isa basis of E,wenotedthatthefamily(u1⇤,...,un⇤)isa basis of the dual space E⇤,

Let us see how the coordinates of a linear form '⇤ E⇤ 2 over the basis (u1⇤,...,un⇤)varyunderachangeofbasis.

Let (u1,...,un)and(v1,...,vn)betwobasesofE,and let P =(aij)bethechangeofbasismatrixfrom(u1,...,un) to (v1,...,vn), so that n

vj = aijui. i=1 X If n n

'⇤ = 'iui⇤ = 'i0vi⇤, i=1 i=1 X X after some algebra, we get

'j0 = aij'i. i=1 X 3.3. THE DUAL SPACE E⇤ AND LINEAR FORMS 213 Comparing with the change of basis n

vj = aijui, i=1 X we note that this time, the coordinates ('i)ofthelinear form '⇤ change in the same direction as the change of basis.

For this reason, we say that the coordinates of linear forms are covariant.

By abuse of language, it is often said that linear forms are covariant,whichexplainswhythetermcovector is also used for a linear form.

Observe that if (e1,...,en)isabasisofthevectorspace E,then,asalinearmapfromE to K,everylinearform f E⇤ is represented by a 1 n matrix, that is, by a row2 vector ⇥ ( ), 1 ··· n with respect to the basis (e1,...,en)ofE,and1ofK, where f(ei)=i. 214 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY n Avectoru = i=1 uiei E is represented by a n 1 matrix, that is, by a column2 vector ⇥ P

u1 . , 0 1 un and the action of f on u@,namelyA f(u), is represented by the matrix product

u1 . 1 n . = 1u1 + + nun. ··· 0u 1 ··· n @ A

On the other hand, with respect to the dual basis (e1⇤,...,en⇤) of E⇤,thelinearformf is represented by the column vector

1 . . 0 1 n @ A 3.3. THE DUAL SPACE E⇤ AND LINEAR FORMS 215 We will now pin down the relationship between a vector space E and its bidual E⇤⇤.

Proposition 3.15. Let E be a vector space. The following properties hold:

(a) The linear map eval : E E⇤⇤ deﬁned such that E ! eval (v)=eval, for all v E, E v 2 that is, eval (v)(u⇤)= u⇤,v = u⇤(v) for every E h i u⇤ E⇤, is injective. 2 (b) When E is of ﬁnite dimension n, the linear map evalE : E E⇤⇤ is an isomorphism (called the canonical! isomorphism).

When E is of ﬁnite dimension and (u1,...,un)isabasis of E,inviewofthecanonicalisomorphism eval : E E⇤⇤,thebasis(u⇤⇤,...,u⇤⇤)ofthebidualis E ! 1 n identiﬁed with (u1,...,un).

Proposition 3.15 can be reformulated very fruitfully in terms of pairings. 216 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY Deﬁnition 3.8. Given two vector spaces E and F over K,apairing between E and F is a bilinear map ': E F K.Suchapairingisnondegenerate i↵ ⇥ ! (1) for every u E,if'(u, v)=0forallv F ,then u =0,and 2 2 (2) for every v F ,if'(u, v)=0forallu E,then v =0. 2 2 Apairing': E F K is often denoted by , : E F ⇥ K!. h i ⇥ !

For example, the map , : E⇤ E K deﬁned earlier is a nondegenerateh pairingi (use⇥ the! proof of (a) in Proposition 3.15).

Given a pairing ': E F K,wecandeﬁnetwomaps ⇥ ! l : E F ⇤ and r : F E⇤ as follows: ' ! ' ! 3.3. THE DUAL SPACE E⇤ AND LINEAR FORMS 217

For every u E,wedefinethelinearforml'(u)inF ⇤ such that 2 l (u)(y)='(u, y)foreveryy F, ' 2 and for every v F ,wedefinethelinearformr (v)in 2 ' E⇤ such that r (v)(x)='(x, v)foreveryx E. ' 2 Proposition 3.16. Given two vector spaces E and F over K, for every nondegenerate pairing ': E F K between E and F , the maps ⇥ ! l' : E F ⇤ and r' : F E⇤ are linear and injective. Furthermore,! if E and! F have finite dimension, then this dimension is the same and l : E F ⇤ and ' ! r : F E⇤ are bijections. ' !

When E has ﬁnite dimension, the nondegenerate pairing , : E⇤ E K yields another proof of the h i ⇥ ! existence of a natural isomorphism between E and E⇤⇤.

Interesting nondegenerate pairings arise in exterior algebra. 218 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY

ME,-rt< Ie. Cl-OCK

Figure 3.4: Metric Clock 3.4. HYPERPLANES AND LINEAR FORMS 219 3.4 Hyperplanes and Linear Forms

Actually, Proposition 3.17 below follows from parts (c) and (d) of Theorem 3.14, but we feel that it is also interesting to give a more direct proof.

Proposition 3.17. Let E be a vector space. The following properties hold:

(a) Given any nonnull linear form f ⇤ E⇤, its kernel 2 H =Kerf ⇤ is a hyperplane. (b) For any hyperplane H in E, there is a (nonnull) linear form f ⇤ E⇤ such that H =Kerf ⇤. 2 (c) Given any hyperplane H in E and any (nonnull) linear form f ⇤ E⇤ such that H =Kerf ⇤, for 2 every linear form g⇤ E⇤, H =Kerg⇤ i↵ g⇤ = f ⇤ for some =0in K2. 6

We leave as an exercise the fact that every subspace V = E of a vector space E,istheintersectionofall hyperplanes6 that contain V .

We now consider the notion of transpose of a linear map and of a matrix. 220 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY 3.5 Transpose of a Linear Map and of a Matrix

Given a linear map f : E F ,itispossibletodeﬁnea ! map f > : F ⇤ E⇤ which has some interesting properties. !

Deﬁnition 3.9. Given a linear map f : E F ,the ! transpose f > : F ⇤ E⇤ of f is the linear map deﬁned such that !

f >(v⇤)=v⇤ f, for every v⇤ F ⇤. 2

Equivalently, the linear map f > : F ⇤ E⇤ is deﬁned such that !

v⇤, f(u) = f >(v⇤),u , h i h i for all u E and all v⇤ F ⇤. 2 2 3.5. TRANSPOSE OF A LINEAR MAP AND OF A MATRIX 221 It is easy to verify that the following properties hold:

(f + g)> = f > + g>

(g f)> = f > g> idE> =idE⇤. Note the reversal of composition on the right-hand side of (g f)> = f > g>.

The equation (g f)> = f > g> implies the following useful proposition.

Proposition 3.18. If f : E F is any linear map, then the following properties! hold:

(1) If f is injective, then f > is surjective.

(2) If f is surjective, then f > is injective. 222 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY We also have the following property showing the natural- ity of the eval map.

Proposition 3.19. For any linear map f : E F , we have !

f >> eval =eval f, E F or equivalently, the following diagram commutes:

f >> / EO⇤⇤ F O⇤⇤

evalE evalF

/ E f F. 3.5. TRANSPOSE OF A LINEAR MAP AND OF A MATRIX 223

If E and F are ﬁnite-dimensional, then evalE and evalF are isomorphisms, so Proposition 3.19 shows that

1 f >> =eval f eval . ( ) F E ⇤ The above equation is often interpreted as follows: if we identify E with its bidual E⇤⇤ and F with its bidual F ⇤⇤, then f >> = f.

This is an abuse of notation; the rigorous statement is ( ). ⇤

Proposition 3.20. Given a linear map f : E F , for any subspace V of E, we have !

0 1 0 0 f(V ) =(f >) (V )= w⇤ F ⇤ f >(w⇤) V . { 2 | 2 }

As a consequence,

0 0 Ker f > =(Imf) and Ker f =(Imf >) . 224 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY The following theorem shows the relationship between the rank of f and the rank of f >.

Theorem 3.21. Given a linear map f : E F , the following properties hold. !

(a) The dual (Im f)⇤ of Im f is isomorphic to Im f > = f >(F ⇤); that is,

(Im f)⇤ Im f >. ⇡ (b) If F is ﬁnite dimensional, then rk(f)=rk(f >).

The following proposition can be shown, but it requires a generalization of the duality theorem.

Proposition 3.22. If f : E F is any linear map, then the following identities hold:! 0 Im f > =(Ker(f)) 0 Ker (f >)=(Imf) 0 Im f =(Ker(f >) 0 Ker (f)=(Imf >) . 3.5. TRANSPOSE OF A LINEAR MAP AND OF A MATRIX 225 The following proposition shows the relationship between the matrix representing a linear map f : E F and the ! matrix representing its transpose f > : F ⇤ E⇤. !

Proposition 3.23. Let E and F be two vector spaces, and let (u1,...,un) be a basis for E, and (v1,...,vm) be a basis for F . Given any linear map f : E F , if M(f) is the m n-matrix representing f !w.r.t. the bases (u ,...,u⇥) and (v ,...,v ), the n m- 1 n 1 m ⇥ matrix M(f >) representing f > : F ⇤ E⇤ w.r.t. the ! dual bases (v1⇤,...,vm⇤ ) and (u1⇤,...,un⇤) is the transpose M(f)> of M(f).

We now can give a very short proof of the fact that the rank of a matrix is equal to the rank of its transpose.

Proposition 3.24. Given a m n matrix A over a ⇥ ﬁeld K, we have rk(A)=rk(A>).

Thus, given an m n-matrix A,themaximumnumber of linearly independent⇥ columns is equal to the maximum number of linearly independent rows. 226 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY Proposition 3.25. Given any m n matrix A over a ⇥ ﬁeld K (typically K = R or K = C), the rank of A is the maximum natural number r such that there is an invertible r r submatrix of A obtained by selecting r rows and r⇥columns of A.

For example, the 3 2matrix ⇥ a11 a12 A = a a 0 21 221 a31 a32 has rank 2 i↵one of the three@ 2 A2matrices ⇥

a11 a12 a11 a12 a21 a22 a a a a a a ✓ 21 22◆✓31 32◆✓31 32◆ is invertible. We will see in Chapter 5 that this is equivalent to the fact the determinant of one of the above matrices is nonzero.

This is not a very ecient way of ﬁnding the rank of amatrix.Wewillseethattherearebetterwaysusing various decompositions such as LU, QR, or SVD. 3.5. TRANSPOSE OF A LINEAR MAP AND OF A MATRIX 227

OF IH1S.1 S IHkf rf '5 OAJL'f OF IMPoRTArJc.E:., AND 1FfE:RE. IS NO WAy rf cAN SS or MY pgAC:lltAL use. \\

Figure 3.5: Beauty 228 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY 3.6 The Four Fundamental Subspaces

Given a linear map f : E F (where E and F are ﬁnite-dimensional), Proposition! 3.20 revealed that the four spaces

Im f, Im f >, Ker f, Ker f > play a special role. They are often called the fundamental subspaces associated with f.

These spaces are related in an intimate manner, since Proposition 3.20 shows that

0 Ker f =(Imf >) 0 Ker f > =(Imf) , and Theorem 3.21 shows that

rk(f)=rk(f >). 3.6. THE FOUR FUNDAMENTAL SUBSPACES 229 It is instructive to translate these relations in terms of matrices (actually, certain linear algebra books make a big deal about this!).

If dim(E)=n and dim(F )=m,givenanybasis(u1,..., un)ofE and a basis (v1,...,vm)ofF ,weknowthatf is represented by an m n matrix A =(a ), where the jth ⇥ ij column of A is equal to f(uj)overthebasis(v1,...,vm).

Furthermore, the transpose map f > is represented by the n m matrix A> (with respect to the dual bases). ⇥ Consequently, the four fundamental spaces

Im f, Im f >, Ker f, Ker f > correspond to 230 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY (1) The column space of A,denotedbyImA or (A); R this is the subspace of Rm spanned by the columns of A,whichcorrespondstotheimageImf of f. (2) The kernel or nullspace of A,denotedbyKerA or (A); this is the subspace of Rn consisting of all N vectors x Rn such that Ax =0. 2 (3) The row space of A,denotedbyImA> or (A>); R this is the subspace of Rn spanned by the rows of A, or equivalently, spanned by the columns of A>,which corresponds to the image Im f > of f >. (4) The left kernel or left nullspace of A denoted by Ker A> or (A>); this is the kernel (nullspace) of A>, N the subspace of Rm consisting of all vectors y Rm 2 such that A>y =0,orequivalently,y>A =0.

Recall that the dimension r of Im f,whichisalsoequal to the dimension of the column space Im A = (A), is the rank of A (and f). R 3.6. THE FOUR FUNDAMENTAL SUBSPACES 231 Then, some our previous results can be reformulated as follows:

1. The column space (A)ofA has dimension r. R 2. The nullspace (A)ofA has dimension n r. N 3. The row space (A>)hasdimensionr. R 4. The left nullspace (A>)ofA has dimension m r. N

The above statements constitute what Strang calls the Fundamental Theorem of Linear Algebra, Part I (see Strang [32]). 232 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY The two statements 0 Ker f =(Imf >) 0 Ker f > =(Imf) translate to (1) The nullspace of A is the orthogonal of the row space of A. (2) The left nullspace of A is the orthogonal of the column space of A.

The above statements constitute what Strang calls the Fundamental Theorem of Linear Algebra, Part II (see Strang [32]).

Since vectors are represented by column vectors and linear forms by row vectors (over a basis in E or F ), a vector x Rn is orthogonal to a linear form y if 2 yx =0. 3.6. THE FOUR FUNDAMENTAL SUBSPACES 233 Then, a vector x Rn is orthogonal to the row space of A i↵ x is orthogonal2 to every row of A,namely Ax =0,whichisequivalenttothefactthatx belong to the nullspace of A.

Similarly, the column vector y Rm (representing a 2 linear form over the dual basis of F ⇤)belongstothe nullspace of A> i↵ A>y =0,i↵y>A =0,whichmeans that the linear form given by y> (over the basis in F )is orthogonal to the column space of A.

Since (2) is equivalent to the fact that the column space of A is equal to the orthogonal of the left nullspace of A,wegetthefollowingcriterionforthesolvabilityofan equation of the form Ax = b:

The equation Ax = b has a solution i↵for all y Rm,if 2 A>y =0,theny>b =0. 234 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY Indeed, the condition on the right-hand side says that b is orthogonal to the left nullspace of A,thatis,thatb belongs to the column space of A.

This criterion can be cheaper to check that checking di- rectly that b is spanned by the columns of A.Forexam- ple, if we consider the system

x x = b 1 2 1 x x = b 2 3 2 x x = b 3 1 3 which, in matrix form, is written Ax = b as below:

1 10 x b 1 1 01 1 x2 = b2 , 0 10 11 0x 1 0b 1 3 3 @ A @ A @ A we see that the rows of the matrix A add up to 0. 3.6. THE FOUR FUNDAMENTAL SUBSPACES 235 In fact, it is easy to convince ourselves that the left nullspace of A is spanned by y =(1, 1, 1), and so the system is solv- able i↵ y>b =0,namely

b1 + b2 + b3 =0. Note that the above criterion can also be stated negatively as follows:

The equation Ax = b has no solution i↵there is some m y R such that A>y =0andy>b =0. 2 6 236 CHAPTER 3. DIRECT SUMS, AFFINE MAPS, THE DUAL SPACE, DUALITY

.!(

------

Figure 3.6: Brain Size?