Section 3 Iterative Methods in Matrix Algebra Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 155 Vector norm Definition A vector norm on Rn, denoted by , is a mapping from Rn to k · k R such that I x 0 for all x Rn, k k ≥ ∈ I x = 0 if and only if x = 0, k k I αx = α x for all α R and x Rn, k k | |k k ∈ ∈ I x + y x + y for all x, y R. k k ≤ k k k k ∈ Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 156 Vector norm Definition (lp norms) The lp (sometimes Lp or `p) norm of a vector is defined by n 1/p 1 p < : x = x p ≤ ∞ k kp | i | Xi=1 p = : x = max xi ∞ k k∞ 1 i n | | ≤ ≤ In particular, the l2 norm is also called the Euclidean norm. Note that when 0 p < 1, is not norm, strictly speaking, ≤ k · kp but have some usages in specific applications. Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 157 7.1 Norms of Vectors and Matrices 433 Note that each of these norms reduces to the absolute value in the case n 1. = The l2 norm is called the Euclidean norm of the vector xbecause it represents the usual notion of distance from the origin in case xis in R1 R, R2,orR3. For example, the t ≡ l2 norm of the vector x (x1, x2, x3) gives the length of the straight line joining the points = 2 3 (0, 0, 0) and (x1, x2, x3). Figure 7.1 shows the boundary of those vectors in R and R that have l2 norm less than 1. Figure 7.2 is a similar illustration for the l norm. l2 norm ∞ Figure 7.1 x x 2 3 The vectors in the ޒ3 The vectors in ޒ2 first octant of with l2 norm less with l2 norm less than 1 are inside (0, 1) than 1 are inside this figure. (0, 0, 1) this figure. (Ϫ1, 0) (1, 0) x 1 (1, 0, 0) (0, 1, 0) x x 1 2 (0, Ϫ1) Figure 7.2 x x Numerical Analysis II – Xiaojing2 Ye, Math & Stat, Georgia State University3 158 (0, 1) (1, 1) (Ϫ1, 1) (0, 0, 1) (1, 0, 1) (0, 1, 1) (1, 1, 1) x 1 (Ϫ1, 0) (1, 0) (1, 0, 0) (0, 1, 0) x 1 x 2 (Ϫ1, Ϫ1) (0, Ϫ1) (1, Ϫ1) (1, 1, 0) The vectors in ޒ2 with The vectors in the first 3 lϱ norm less than 1 are octant of ޒ with lϱ norm inside this figure. less than 1 are inside this figure. Copyright 2010 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part. Due to electronic rights, some third party content may be suppressed from the eBook and/or eChapter(s). Editorial review has deemed that any suppressed content does not materially affect the overall learning experience. Cengage Learning reserves the right to remove additional content at any time if subsequent rights restrictions require it. 7.1 Norms of Vectors and Matrices 433 Note that each of these norms reduces to the absolute value in the case n 1. = The l2 norm is called the Euclidean norm of the vector xbecause it represents the usual notion of distance from the origin in case xis in R1 R, R2,orR3. For example, the t ≡ l2 norm of the vector x (x1, x2, x3) gives the length of the straight line joining the points = 2 3 (0, 0, 0) and (x1, x2, x3). Figure 7.1 shows the boundary of those vectors in R and R that have l2 norm less than 1. Figure 7.2 is a similar illustration for the l norm. ∞ Figure 7.1 x x 2 3 The vectors in the ޒ3 The vectors in ޒ2 first octant of with l2 norm less with l2 norm less than 1 are inside (0, 1) than 1 are inside this figure. (0, 0, 1) this figure. (Ϫ1, 0) (1, 0) x 1 (1, 0, 0) (0, 1, 0) x x 1 2 Ϫ l norm (0, 1) ∞ Figure 7.2 x 2 x 3 (0, 1) (1, 1) (Ϫ1, 1) (0, 0, 1) (1, 0, 1) (0, 1, 1) (1, 1, 1) x 1 (Ϫ1, 0) (1, 0) (1, 0, 0) (0, 1, 0) x 1 x 2 (Ϫ1, Ϫ1) (0, Ϫ1) (1, Ϫ1) (1, 1, 0) The vectors in ޒ2 with The vectors in the first 3 lϱ norm less than 1 are octant of ޒ with lϱ norm inside this figure. less than 1 are inside this figure. Copyright 2010 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part. Due to electronic rights, some third party content may be suppressed from the eBook and/or eChapter(s). Editorial reviewNumerical has deemed that any Analysis suppressed content II – does Xiaojing not materially Ye, affect Math the overall & learning Stat, experience. Georgia Cengage State Learning University reserves the right to remove additional content at any time if subsequent rights restrictions 159 require it. Vector norms Example 3 Compute the l2 and l norms of vector x = (1, 1, 2) R . ∞ − ∈ Solution: x = 1 2 + 1 2 + 2 2 = √6 k k2 | | | − | | | q x = max xi = max 1 , 1 , 2 = 2 k k∞ 1 i 3 | | {| | | − | | |} ≤ ≤ Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 160 Theorem (Cauchy-Schwarz inequality) n For any vectors x = (x1,..., xn)> R and n ∈ y = (y1,..., yn)> R , there is ∈ n n 1/2 n 1/2 2 2 x>y = x y x y = x y | | i i ≤ | i | | i | k k2k k2 i=1 i=1 i=1 X X X Proof. It is obviously true for x = 0 or y = 0. If x, y = 0, then for any 6 λ R, there is ∈ 2 2 2 2 0 x λy = x 2λx>y + λ y ≤ k − k2 k k2 − k k2 and the equality holds when λ = x / y . k k2 k k2 Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 161 Distance between vectors Definition (Distance between two vectors) n The lp distance (1 p ) between two vectors x, y R is ≤ ≤ ∞ ∈ defined by x y . k − kp Definition (Convergence of a sequence of vectors) A sequence x(k) is said to converge with respect to the l { } p norm if for any given > 0, there exists an integer N() such that x(k) x < , for all k N() k − k ≥ Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 162 Convergence of a sequence of vectors Theorem A sequence of vectors x(k) converges to x if and only if { } x(k) x for every i = 1, 2,..., n. i → i Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 163 Theorem For any vector x Rn, there is ∈ x x 2 √n x k k∞ ≤ k k ≤ k k∞ Proof. 2 2 2 x = max xi = max xi x1 + + xn = x 2 k k∞ i | | i | | ≤ | | ··· | | k k q 2 q 2 2 x 2 = x1 + + xn n max xi k k | | ··· | | ≤ i | | q 2 q = √n max xi = √n max xi = √n x i | | i | | k k∞ q Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 164 7.1 Norms of Vectors and Matrices 437 Proof Let xj be a coordinate of xsuch that x max1 i n xi xj . Then ∥ ∥∞ = ≤ ≤ | |=| | n 2 2 2 2 2 x xj xj xi x 2, ∥ ∥∞ =| | = ≤ = ∥ ∥ i 1 != and x x 2. ∥ ∥∞ ≤∥ ∥ So n n 2 2 2 2 2 x 2 xi xj nxj n x , ∥ ∥ = ≤ = = || ||∞ i 1 i 1 != != and x 2 √n x . ∥ ∥ ≤ ∥ ∥∞ Figure 7.3 illustrates this result when n 2. = 2 Compare l2 and l norms in R Figure 7.3 ∞ x 2 ʈxʈϱ р 1 1 ʈxʈ2 р 1 Ϫ 11x 1 2 x р √ ʈ ʈϱ 2 Ϫ1 Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 165 Example 4 In Example 3, we found that the sequence x(k) , defined by { } t (k) 1 3 k x 1, 2 , , e− sin k , = + k k2 " # converges to x (1, 2, 0, 0)t with respect to the l norm. Show that this sequence also = ∞ converges to xwith respect to the l2 norm. Solution Given any ε > 0, there exists an integer N(ε/2) with the property that ε x(k) x < , ∥ − ∥∞ 2 whenever k N(ε/2). By Theorem 7.7, this implies that ≥ (k) (k) x x 2 √4 x x 2(ε/2) ε, ∥ − ∥ ≤ ∥ − ∥∞ ≤ = (k) when k N(ε/2). So x also converges to xwith respect to the l2 norm. ≥ { } Copyright 2010 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part. Due to electronic rights, some third party content may be suppressed from the eBook and/or eChapter(s). Editorial review has deemed that any suppressed content does not materially affect the overall learning experience. Cengage Learning reserves the right to remove additional content at any time if subsequent rights restrictions require it. Matrix norm Definition A matrix norm on the set of n n matrices is a real-valued × function, denoted by , that satisfies the follows for all n n k · k A, B R × and α R: ∈ ∈ I A 0 k k ≥ I A = 0 if and only if A = 0 the zero matrix, k k I αA = α A k k | |k k I A + B A + B k k ≤ k k k k I AB A B k k ≤ k kk k Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 166 Distance between matrices Definition n n Suppose is a norm defined on R × .