Recitation 4 Carles Domingo Fall 2020 Norms Definition (Norm) A norm k · k on V is a function k · k : V → R≥0 that verifies 1. Positive definiteness: if kvk = 0 =⇒ v = 0. 2. Homogeneity: kαvk = |α| × kvk 3. Triangle inequality: ku + vk ≤ kuk + kvk A norm defines a distance d(x, y) = kx − yk on the vector space V . Definition (Distance) A distance d on a set S (not necessarily a vector space) is a function d : S × S → R. 1. Positive definiteness: if d(x, y) ≥ 0 for all x, y ∈ S and d(x, y) ≥ 0 i x = y. 2. Symmetry: d(x, y) = d(y, x) for all x, y ∈ S. 3. Triangle inequality: d(x, y) ≤ d(x, z) + d(z, y) 2/14 Inner Products Definition (Inner product) Let V be a vector space. An inner product on V is a function h·, ·i from V × V to R that verifies the following points: 1. Symmetry: hu, vi = hv, ui for all u, v ∈ V . 2. Linearity: hu + v, wi = hu, wi + hv, wi and hαv, wi = αhv, wi for all u, v, w ∈ V and α ∈ R. 3. Positive definiteness: hv, vi ≥ 0 with equality if and only if v = 0. Important: An inner product defines a norm kuk = phu, ui. Most used inner product: the Euclidean inner product: > hu, vi =√u v. The corresponding norm is > √ Pn 2 kuk = u u = i=1 ui . Given an inner product, we can define the angle between two vectors: cos(θ) = hu,vi . kukkvk 3/14 Questions: Norms & Inner Products 3 1. Which of the following functions are inner products for x, y ∈ R ? i. f(x, y) = x1y2 + x2y3 + x3y1 2 2 2 2 2 2 ii. f(x, y) = x1y1 + x2y2 + x1y1 iii. f(x, y) = x1y1 + x3y3 4/14 Questions: Norms & Inner Products 3 1. Which of the following functions are inner products for x, y ∈ R ? i. f(x, y) = x1y2 + x2y3 + x3y1 2 2 2 2 2 2 ii. f(x, y) = x1y1 + x2y2 + x1y1 iii. f(x, y) = x1y1 + x3y3 4/14 Questions: Norms & Inner Products 3 1. Which of the following functions are inner products for x, y ∈ R ? i. f(x, y) = x1y2 + x2y3 + x3y1 2 2 2 2 2 2 ii. f(x, y) = x1y1 + x2y2 + x1y1 iii. f(x, y) = x1y1 + x3y3 4/14 Questions: Norms & Inner Products m×n n 2. For A ∈ R and x ∈ R , prove that v u m n uX X 2 kAxk ≤ kxkt Ai,j i=1 j=1 5/14 Questions: Norms & Inner Products m×n n 2. For A ∈ R and x ∈ R , prove that v u m n uX X 2 kAxk ≤ kxkt Ai,j i=1 j=1 5/14 Questions: Orthogonality Recall from the lecture: 1. Two vectors u, v are orthogonal if hu, vi = 0. 2. If U is a subspace of of a vector space V with inner product h, i, the orthogonal projection PU : V → U is defined as x 7→ arg minu∈U kx − uk. Exercises: 1. Let v1, ..., vk be a list of orthogonal vectors. Show that v1, ..., vk are linearly independent. n 2. Let U be the subspace of R with orthonormal basis u1, ..., uk. i. Prove that the orthogonal projection of v ∈ Rn onto U can be Pk expressed as PU (v) = i=0hv, uiiui. (Use the fact that the orthonormal basis for a subspace of R can be extended to obtain an orthonormal basis for R). ii. Prove that kPU (v)k ≤ kvk. iii. Prove that v − PU (v) is orthogonal to PU (v). 6/14 Questions: Orthogonality 1. Let v1, ..., vk be a list of orthogonal vectors. Show that v1, ..., vk are linearly independent. 7/14 Questions: Orthogonality 1. Let v1, ..., vk be a list of orthogonal vectors. Show that v1, ..., vk are linearly independent. 7/14 Questions: Orthogonality n 2. Let U be the subspace of R with orthonormal basis u1, ..., uk. n i. Prove that the orthogonal projection of v ∈ R onto U can be Pk expressed as PU (v) = i=0hv, uiiui. (Use the fact that the orthonormal basis for a subspace of R can be extended to obtain an orthonormal basis for R). ii. Prove that kPU (v)k ≤ kvk. iii. Prove that v − PU (v) is orthogonal to PU (v). 8/14 Questions: Orthogonality n 2. Let U be the subspace of R with orthonormal basis u1, ..., uk. n i. Prove that the orthogonal projection of v ∈ R onto U can be Pk expressed as PU (v) = i=0hv, uiiui. (Use the fact that the orthonormal basis for a subspace of R can be extended to obtain an orthonormal basis for R). ii. Prove that kPU (v)k ≤ kvk. iii. Prove that v − PU (v) is orthogonal to PU (v). 8/14 Questions: Orthogonality n 2. Let U be the subspace of R with orthonormal basis u1, ..., uk. n i. Prove that the orthogonal projection of v ∈ R onto U can be Pk expressed as PU (v) = i=0hv, uiiui. (Use the fact that the orthonormal basis for a subspace of R can be extended to obtain an orthonormal basis for R). ii. Prove that kPU (v)k ≤ kvk. iii. Prove that v − PU (v) is orthogonal to PU (v). 8/14 Questions: Orthogonality n 2. Let U be the subspace of R with orthonormal basis u1, ..., uk. n i. Prove that the orthogonal projection of v ∈ R onto U can be Pk expressed as PU (v) = i=0hv, uiiui. (Use the fact that the orthonormal basis for a subspace of R can be extended to obtain an orthonormal basis for R). ii. Prove that kPU (v)k ≤ kvk. iii. Prove that v − PU (v) is orthogonal to PU (v). 8/14 Questions: Orthogonality n 2. Let U be the subspace of R with orthonormal basis u1, ..., uk. n i. Prove that the orthogonal projection of v ∈ R onto U can be Pk expressed as PU (v) = i=0hv, uiiui. (Use the fact that the orthonormal basis for a subspace of R can be extended to obtain an orthonormal basis for R). ii. Prove that kPU (v)k ≤ kvk. iii. Prove that v − PU (v) is orthogonal to PU (v). 8/14 Questions: Orthogonal Complement 1. Let S, U be subspaces of a vector space V . Prove the following statement: S ⊂ U =⇒ S⊥ ⊃ U ⊥ n×m 2. Let A ∈ R . Assume the Euclidean inner product. Prove that Im(AT ) = Ker(A)⊥. (Hint: =⇒ is easy. Use (1) for ⇐= ) 9/14 Questions: Orthogonal Complement 1. Let S, U be subspaces of a vector space V . Prove the following statement: S ⊂ U =⇒ S⊥ ⊃ U ⊥ 10/14 Questions: Orthogonal Complement n×m 2. Let A ∈ R . Assume the Euclidean inner product. Prove that Im(AT ) = Ker(A)⊥. (Hint: =⇒ is easy. Use (1) for ⇐= ) 11/14 Questions: Orthogonal Complement n×m 2. Let A ∈ R . Assume the Euclidean inner product. Prove that Im(AT ) = Ker(A)⊥. (Hint: =⇒ is easy. Use (1) for ⇐= ) 11/14 Questions: Idempotence Definition (Idempotence) An matrix P is idempotent when P 2 = P . 1. Show that X(XT X)−1XT is idempotent. 2. Show that all orthogonal projections are idempotent. 3. Give an example of an idempotent matrix that is not an orthogonal projection. (Hint: Show that your matrix does not minimize the distance to subspace it projects onto.) 12/14 Questions: Idempotence 1. Show that X(XT X)−1XT is idempotent. 2. Show that all orthogonal projections are idempotent. 13/14 Questions: Idempotence 1. Show that X(XT X)−1XT is idempotent. 2. Show that all orthogonal projections are idempotent. 13/14 Questions: Idempotence 3. Give an example of an idempotent matrix that is not an orthogonal projection. (Hint: Show that your matrix does not minimize the distance to subspace it projects onto.) 14/14 Questions: Idempotence 3. Give an example of an idempotent matrix that is not an orthogonal projection. (Hint: Show that your matrix does not minimize the distance to subspace it projects onto.) 14/14.