Unit 2: Projections and subspaces Juan Luis Melero and Eduardo Eyras September 2018 1 Contents 1 Operations and definitions3 1.1 Inner product (scalar product)..................3 1.2 Norm of a vector.........................3 1.3 Distance between vectors.....................5 1.4 Angle between two vectors....................6 1.5 Orthogonal projection......................7 2 Gram-Schmidt orthogonalization9 2.1 Process of orthogonalization................... 10 2.2 Example of orthogonal basis in R2 ................ 11 2.3 Example of orthogonal basis in R3 ................ 12 3 Vector subspaces 13 3.1 Vector space projection...................... 14 4 Exercises 15 5 R practical 17 5.1 Inner product........................... 17 5.2 Norm of a vector......................... 17 5.3 Distance between two vectors.................. 17 5.4 Angle between two vectors.................... 18 5.5 Orthogonal projection...................... 18 5.6 Gram-Schmidt orthogonalization................ 20 2 1 Operations and definitions 1.1 Inner product (scalar product) The inner product (also called dot product or scalar product) is an operation that, taking two vectors of the same dimension, returns a scalar (real) value. We define an inner product as: h:; :i : V × V ! R (~u;~v) !h~u;~vi 2 R with the following properties: 1. h~u;~vi = h~v; ~ui Commutative property 2. h~u;~v + ~wi = h~u + ~v; ~u + ~wi Distributive property with respect to the vector addition 3. c h~u;~vi = hc~u;~vi = h~u;c~vi 8c 2 R Associative property with re- spect to the scalar multiplication. 4. h~u;~ui ≥ 0 Non-negative property 5. h~u;~ui = 0 () ~u = ~0 The standard definition of a scalar product in Rn is the following: 0 1 0 1 u1 v1 B . C B . C ~u = @ . A ;~v = @ . A un vn 0 1 v1 t u : : : u B . C h~u;~vi = ~u ~v = 1 n @ . A = u1v1 + ··· + unvn vn 1.2 Norm of a vector The norm of the vector can be considered as the length of the vector. The concept of distance exists in any number of dimensions n and we use the norm of a vector to define it. The definition of the norm is the square root of the inner product of a vector with itself: 3 Norm = k~uk = ph~u;~ui The norm of a vector has the following properties: 1. k~uk = 0 () ~u = 0 Only the vector ~0 has "0" length. 2. k~uk = 1 () h~u;~ui = 1 Definition of unit vector (vector of lenght 1) 3. kc~uk = jcjk~uk 1 4. 8~v 2 V; k~vk= 6 0 =) k~vk~v is a unit vector. Proof of property 3: kc~uk = phc~u;c~ui = pc2 h~u;~ui = jcjph~u;~ui = jcjk~uk Proof of property 4: s s 1 1 1 1 1 ~v == ~v; ~v = h~v;~vi = k~vk = 1 k~vk k~vk k~vk k~vk2 k~vk The length (module) of a vector in Rn can be written as: v 0 1 u u 1 p u 1 n ! 2 p u . p X 2 k~uk = h~v;~vi = ~ut~u = u u : : : u B . C = u u + ··· + u u = u t 1 n @ . A 1 1 n n i i=1 un This is called the L2 Euclidean norm, but there are other possible definitions of norm: 1 n ! p X p Lp norm k~ukp = ui i=1 n X L1 norm k~uk = juij i=1 L1 norm k~uk1 = max fjuijg i 4 1.3 Distance between vectors As mentioned before, the norm allows us to define a distance in a Vector Space. The mathematical definition of the distance d between two vectors ~u;~v is the following: d: V × V ! R (~u;~v) !d(~u;~v) = k~u − ~vk 2 R A distance has the following properties: 1. d(~u;~v) ≥ 0 Non-negative 2. d(~u;~v) = d(~v; ~u) Symmetric 3. d(~u;~v) = 0 () ~u = ~v "Identity" 4. d(~u;~v) + d(~u;~w) ≥ d(~u;~w) Triangle inequality To proof the triangle inequality, first we need to know the Cauchy-Schwartz inequality: jh~u;~vij ≤ k~uk · k~vk Proof: h~u;~vi Consider: ~u;~v 2 V; ~u;~v 6= 0; λ = k~vk2 We define a new vector ~z = ~u − λ~v 0 ≤ k~zk = h~z; ~zi (this is always true for any vector) 0 ≤ h~u − λ~v; ~u − λ~vi = h~u;~ui − λ h~v; ~ui − λ h~u;~vi + λ2 h~v;~vi = h~u;~vi h~u;~vi2 = k~uk2 − 2λ h~u;~vi + λ2k~vk2 = k~uk2 − 2 h~u;~vi + = k~vk2 k~vk2 h~u;~vi2 h~u;~vi2 k~uk2 − ! k~uk ≥ ! k~ukk~vk ≥ jh~u;~vij k~vk2 k~vk2 5 Proof of the triangle inequality: k~u + ~vk ≤ k~uk + k~vk k~u + ~vk2 = h~u + ~v; ~u + ~vi = h~u;~vi + 2 h~u;~vi + h~v;~vi = = k~uk2 + k~vk2 + 2 h~u;~vi ≤ ≤ k~uk2 + k~vk2 + 2k~ukk~vk = (k~uk + k~vk)2 Example in R2: ~ ~ ~a1 = (2; 2); b1 = (1; 2:5);~a2 = ~a1 − b1 = (1; −0:5) r ~ ~ D ~ ~ E d(~a1; b1) = k~a1 − b1k = ~a1 − b1;~a1 − b1 = p = p(2 − 1)2 + (2 − 2:5)2 = p(1)2 + (0:5)2 = 1:25 Another way to do that is with the vector ”difference”, in this case, ~a2: p p ~ ~ p p d(~a1; b1) = k~a1−b1k = k~a2k = h~a2;~a2i = h(1; −0:5); (1; −0:5)i = 1 + 0:25 = 1:25 1.4 Angle between two vectors We define the angle between two vectors as: h~u;~vi h~u;~vi cos θ = ! θ = arccos k~ukk~vk k~ukk~vk The relationships and properties of the inner product with the angle are: π h~u;~vi = 0 () θ = Orthogonal vectors 2 h~u;~vi = k~ukk~vk () θ = 0 Parallel vectors h~u;~vi = −k~ukk~vk () θ = π Anti-parallel vectors 6 Using the properties of the angle between two vectors, the Pythagorean the- orem can be proven: if ~u ? ~v =) k~u + ~vk2 = k~uk2 + k~vk2 k~u + ~vk2 = h~u + ~v; ~u + ~vi = h~u;~vi + 2 h~u;~vi + h~v;~vi = = k~uk2 + k~vk2 + 2 h~u;~vi = k~uk2 + k~vk2 (if ~u ? ~v =) h~u;~vi = 0) With the angle definition we can also proof the Cauchy-Schwartz inequality in a more simple way: h~u;~vi cos θ = ! h~u;~vi = k~ukk~vk cos θ k~ukk~vk Since jcos θj ≤ 1 ! jh~u;~vij ≤ k~ukk~vk 1.5 Orthogonal projection Consider a vector space V over the real numbers R. Given two vectors ~u;~v 2 V with different directions we can define a orthogonal projection of ~u onto ~v as: h~u;~vi P roj (~u) = ~v ~v k~vk2 In Figure1 we can see a projection in R2. Figure 1: Projection of ~u onto ~v in R2. 7 We can define the projection as the components of ~u that are in the direction of ~v. There are some properties related to the projection: 1. The length of ~v does not contribute to the definition projection. 2. ~v marks the direction of the projection. Proof of property 1: h~u;~vi k~ukk~vk cos θ ~v P roj (~u) = ~v = ~v = k~uk cos θ ~v k~vk2 k~vk2 k~vk ~v k~vk is a unit vector for any ~v. Hence, only k~uk, cos θ, and the direction of contribute to the definition. h~u;~vi Proof of property 2: k~vk2 is a scalar, i.e. a real number. Therefore, we can rewrite the projection as h~u;~vi P roj (~u) = ~v = α~v ~v k~vk2 Theorem:(~u − P roj~v(~u)) ? P roj~v(~u) Proof: h~u;~vi h~u;~vi h~u;~vi h~u;~vi h~u;~vi h~u − P roj (~u); P roj (~u)i = ~u − ~v; ~v = ~u; ~v − ~v; ~v = ~v ~v k~vk2 k~vk2 k~vk2 k~vk2 k~vk2 h~u;~vi h~u;~vi2 1 1 h~u;~vi − h~v;~vi = h~u;~vi2 − k~vk2 h~u;~vi2 = 0 k~vk2 k~vk2 k~vk2 k~vk4 As mentioned above, P roj~v(~u) = α~v. Using the condition of orthogonality ~u−P roj~v(~u)) ? P roj~v(~u) ! hα~v − ~u;α~vi = 0, We can recover the definition of orthogonal projection: 8 0 = hα~v − ~u;α~vi = α2 h~v;~vi − α h~u;~vi ! α (α h~v;~vi − h~u;~vi) = 0 h~u;~vi α = 0 α = h~v;~vi h~u;~vi P roj (~u) = ~v ~v h~v;~vi In general, given a basis of orthogonal vectors B = f~w1; : : : ; ~wng, the repre- sentation of a vector in this basis is given by the projections: X X h~u;~wii u = P roj (~u) = ~w ~wi k~w k i i i i 1 1 Example: consider the following basis in 2, B = ~v = ;~v = , R 1 1 2 −1 a the vector in a basis ~u = can be written as: b a + b 1 a − b 1 ~u = P roj (~u) + P roj (~u) = + ~v1 ~v2 2 1 2 −1 which gives us the representation in the basis B as: 0 a+b 1 2 RepB(~u) = @ A a−b 2 2 Gram-Schmidt orthogonalization The orthogonal projection suggests that we can decompose any vector into orthogonal components.