Numerical Linear Algebra Lecture 3

Larisa Beilina Lecture 3 NLA notions

Norms Sherman-Morrison formula Perturbation theory Condition number, relative condition number Gaussian elimination

Larisa Beilina Lecture 3 Singular values

The singular values of a compact operator T : X Y acting between Hilbert spaces X and Y , are the square roots of the→ eigenvalues of the nonnegative self-adjoint operator T ∗T : X X (where T ∗ denotes the adjoint of T ). For the case of a matrix A, singular→ values are computed as: σ = λ(A∗A). The singularp values are nonnegative real numbers, usually listed in decreasing order (σ1(T ),σ2(T ),...). If T is self-adjoint, then the largest singular value σ1(T ) is equal to the operator norm of T . In the case of a normal matrix A (or A∗A = AA∗, when A is real then AT A = AAT ), the spectral theorem can be applied to obtain unitary diagonalization of A as A = UΛU∗. Therefore, √A∗A = U Λ U∗ and so the singular values are simply the absolute values of the eig|envalues.|

Larisa Beilina Lecture 3 Norm of a vector

We are going to use following such called Lp-norms which are usually denoted by p: k · k n p 1/p x p =( xk ) , p 1. k k | | ≥ Xk=1

n p = 1, x 1 = xk , one-norm k k k=1 | | P n 2 1/2 p = 2, x 2 =( xk ) , two-norm k k k=1 | | P p = , x = max1 k n xk , max-norm or inﬁnity-norm. ∞ k k∞ ≤ ≤ | |

Larisa Beilina Lecture 3 Norm of a vector

All these norms, as all other vector norms, has following properties:

x = 0 x > 0 (positivity), 0 = 0. 6 → k k k k αx = α x for all α R (homogeneity) k k | |k k ∈ x + y x + y (triangle inequality) k k ≤ k k k k

Larisa Beilina Lecture 3 Norm of a vector

Norms are deﬁned diﬀerently, but they can be compared. Two norms α and β on a vector space V are called equivalent if there exist kpositive · k realk · numbers k C and D such that for all x in V

C x x D x . k kα ≤ k kβ ≤ k kα T For example, let x =(x1, ...., xn)

α x 1 x 2 β x 1 k k ≤ k k ≤ k k In this case we have: α = 1/√n,β = 1.

Larisa Beilina Lecture 3 Norm of a vector

Other examples:

x 2 x 1 √n x 2 k k ≤ k k ≤ k k

x x 2 √n x k k∞ ≤ k k ≤ k k∞

x x 1 n x , k k∞ ≤ k k ≤ k k∞

x x 2 x 1 √n x 2 n x k k∞ ≤ k k ≤ k k ≤ k k ≤ k k∞ If the vector space is a ﬁnite-dimensional real or complex one, all norms are equivalent. On the other hand, in the case of inﬁnite-dimensional vector spaces, not all norms are equivalent.

Larisa Beilina Lecture 3 Inner product

The dot product of two vectors x =(x1,..., xn) and y =(y1,..., yn) is deﬁned as:

n T T x y =(x, y)= xk yk = x y, x 2 = √x x. · k=1 k k We note that xPT y is a scalar, but xy T is a matrix. Example 3 x T y =[ 1, 2, 3]  2  =( 1) (3)+ 2 2 + 3 1 = 4. − · 1 − · · ·   1 3 2 1 T − − − − xy =  2  [3, 2, 1]=  6 4 2  3 · 9 6 3    

Larisa Beilina Lecture 3 Norm of a vector

Example

1 − x 1 = 1 + 2 + 3 + 5 = 11  2  x = ; || || |− | | | | | |− | 3 2 2 2 2   x 2 = ( 1) + 2 + 3 +( 5) = √39  5  || || − −  −  p x = max( 1 , 2 , 3 , 5 )= 5. || ||∞ |− | | | | | |− |

Larisa Beilina Lecture 3 Norm of a vector

x A vector x is normalized if x = 1. If x = 0 then is normalized || || 6 x vector. || || Example

1 − T  2  2 x x =[ 1, 2, 3, 5] =( 1) ( 1)+ 2 2 + 3 3 +( 5) = 39 − − · 3 − · − · · −    5   − 

x 2 = √39 || || x 1 2 3 5 T V = = − , , , − = V 2 = 1 x √39 √39 √39 √39 ⇒ || || || ||

Larisa Beilina Lecture 3 Matrix norm. Deﬁnition

Let K will denote the ﬁeld of real or complex numbers. Let K m×n denote the vector space containing all matrices with m rows and n columns with entries in K. Let A∗ denotes the conjugate transpose of matrix A. A matrix norm is a vector norm on K m×n. That is, if A denotes the norm of the matrix A, then, k k

A > 0 if A = 0 and A = 0 if A = 0. k k 6 k k αA = α A for all α in K and all matrices A in K m×n. k k | |k k A + B A + B for all matrices A and B in K m×n. k k ≤ k k k k

Larisa Beilina Lecture 3 Matrix norm. Deﬁnition

In the case of square matrices (thus, m = n), some (but not all) matrix norms satisfy the following condition, which is related to the fact that matrices are more than just vectors: AB A B for all matrices A and B in K n×n. Ak matrixk ≤ k normkk thatk satisﬁes this additional property is called a sub-multiplicative norm. The set of all n-by-n matrices, together with such a sub-multiplicative norm, is an example of a Banach algebra.

Larisa Beilina Lecture 3 Matrix norm. Induced norm

If vector norms on Km and Kn are given (K is ﬁeld of real or complex numbers), then one deﬁnes the corresponding induced norm or operator norm on the space of m-by-n matrices as the following maxima:

A = max Ax : x K n with x = 1 k k {k k ∈ k k } Ax = max k k : x K n with x = 0 . x ∈ 6 k k If m = n and one uses the same norm on the domain and the range, then the induced operator norm is a sub-multiplicative matrix norm.

Larisa Beilina Lecture 3 Matrix norm. p-norm

The operator norm corresponding to the p-norm for vectors is:

Ax p A p = max k k . k k x=0 x 6 k kp In the case of p = 1 and p = , the norms can be computed as: ∞ m A 1 = max aij , k k 1 j n | | ≤ ≤ Xi=1 which is simply the maximum absolute column sum of the matrix.

n A = max aij , k k∞ 1 i m | | ≤ ≤ Xj=1

which is simply the maximum absolute row sum of the matrix

Larisa Beilina Lecture 3 Matrix norm. Example

For example, if the matrix A is deﬁned by

3 5 7 A = 2 6 4 , 0 2 8   then we have A 1 = max(5, 13, 19)= 19 and A = max(||15||, 12, 10)= 15. Consider another example || ||∞ 2 4 2 1 3 1 5 2 A = , 1 2 3 3   0 6 1 2   where we add all the entries in each column and determine the greatest value, which results in A 1 = max(6, 13, 11, 8)= 13. || || We can do the same for the rows and get A ∞ = max(9, 11, 9, 9)= 11. Thus 11 is our max. || ||

Larisa Beilina Lecture 3 Matrix norm. Example

In the special case of p = 2 (the Euclidean norm) and m = n (square matrices), the induced matrix norm is the spectral norm. The spectral norm of a matrix A is the largest singular value of A i.e. the square root of the largest eigenvalue of the positive-semideﬁnite matrix A∗A:

∗ A 2 = λmax(A A)= σmax(A) k k p where A∗ denotes the conjugate transpose of A.

Larisa Beilina Lecture 3 Matrix norm. Example

Example 1 2 3 − − A =  6 4 2  9 6 3  −  A 2 = max λ(AT A) || || p 1 6 9 118 32 36 T T − A =  2 4 6  , A A =  32 56 4  −3 2− 3 −36 4− 22  −   − 

8.9683 max λ(AT A) =max(2.9947, 6.7322, λ(AT A)=  45.3229  ; p 11.9042)= 11.9042 141.7089  

Larisa Beilina Lecture 3 Matrix norm. Example

Example

1 0 1 0 1 λ 0 A = ; AT = ; AT A λI = − = 0; 0 1 0 1 − 0 1 λ −

λ1 = 1, λ2 = 1; A 2 = max λ(AT A) = max(1, 1)= 1 || || p

Larisa Beilina Lecture 3 Matrix norm

Any induced norm satisﬁes the inequality A ρ(A), k k≥ where ρ(A) := max λ1 ,..., λm is the spectral radius of A. For a symmetric or hermitian{| | matrix| A|}, we have equality for the 2-norm, since in this case the 2-norm is the spectral radius of A. For an arbitrary matrix, we may not have equality for any norm. Example Take 0 1 A = , 0 0 the spectral radius ρ(A) of A is 0, but A is not the zero matrix, and so none of the induced norms are equal to the spectral radius of A:

A 1 = 1, A = 1, A 2 = 1. k k k k∞ k k

∗ ∗ 0 0 0 1 0 0 A = λmax(A A)= σmax(A); A A = = 2 1 00 0 0 1 k k p

Larisa Beilina Lecture 3 Matrix norm. "Entrywise" norms

For square matrices we have the spectral radius formula:

1 lim Ar /r = ρ(A). r →∞ k k These vector norms treat an m n matrix as a vector of size m n , × · and use one of the familiar vector norms. For example, using the p-norm for vectors, we get:

1/p m n p A p =  aij  . k k | | Xi=1 Xj=1   The special case p = 2 is the Frobenius norm, and p = yields ∞ the maximum norm.

Larisa Beilina Lecture 3 Matrix norm. Frobenius norm.

For p = 2, this is called the Frobenius norm or the Hilbert - Schmidt norm, though the latter term is often reserved for operators on Hilbert space. This norm can be deﬁned in various ways:

m n min{m, n} 2 ∗ 2 A F = v aij = trace(A A)= v σ k k u | | u i uXi=1 Xj=1 p u Xi=1 t t ∗ where A denotes the conjugate transpose of A, σi are the singular values of A, and the trace function is used. The Frobenius norm is very similar to the Euclidean norm on Kn and comes from an inner product on the space of all matrices. The Frobenius norm is sub-multiplicative and is very useful for numerical linear algebra. This norm is often easier to compute than induced norms and has the useful property of being invariant under rotations.

Larisa Beilina Lecture 3 Matrix norm. Max norm.

The max norm is the elementwise norm with p = : ∞

A max = max aij . k k {| |} This norm is not sub-multiplicative.

Larisa Beilina Lecture 3 Matrix norm. Equivalence of norms.

For any two vector norms and , we have ||·||α ||·||β r A A s A k kα ≤ k kβ ≤ k kα m n for some positive numbers r and s, for all matrices A in K × . In other words, they are equivalent norms; they induce the same m n topology on K × . This is a special case of the equivalence of norms in ﬁnite-dimensional Normed vector spaces.

Larisa Beilina Lecture 3 Examples of norm equivalence

For matrix A Rm n the following inequalities hold: ∈ × A 2 A F √r A 2, where r is the rank of A k k ≤ k k ≤ k k A max A 2 √mn A max k1 k ≤ k k ≤ k k A A 2 √m A √n k k∞ ≤ k k ≤ k k∞ 1 A 1 A 2 √n A 1. √m k k ≤ k k ≤ k k Here, p refers to the matrix norm induced by the vector p-norm. ||·|| Another useful inequality between matrix norms is

A 2 A 1 A . k k ≤ pk k k k∞

Larisa Beilina Lecture 3 Sherman - Morrison formula

Suppose A is an invertible square matrix and u, v are vectors. Suppose furthermore that 1 + v T A 1u = 0. Then the − 6 Sherman-Morrison formula states that

1 T 1 T 1 1 A− uv A− (A + uv )− = A− T 1 . (1) − 1 + v A− u Here, uv T is the outer product of two vectors u and v. If the inverse of A is already known, the formula provides a numerically cheap way to compute the inverse of A corrected by the matrix uv T .

Larisa Beilina Lecture 3 We verify the properties of the inverse. A matrix Y (in this case the right-hand side of the Sherman - Morrison formula) is the inverse of a matrix X (in this case A + uv T ) if and only if XY = YX = I. We ﬁrst verify that the right hand side (Y) satisﬁes XY = I.

1 T 1 T 1 A− uv A− XY =(A + uv ) A− T 1 − 1 + v A− u 1 T 1 T 1 T 1 1 T 1 AA− uv A− + uv A− uv A− = AA− + uv A− T 1 − 1 + v A− u T 1 T 1 T 1 T 1 uv A− + uv A− uv A− = I + uv A− T 1 − 1 + v A− u T 1 T 1 T 1 u(1 + v A− u)v A− = I + uv A− T 1 . − 1 + v A− u

Larisa Beilina Lecture 3 T 1 T 1 Note that v A− u is a scalar, so (1 + v A− u) can be factored out, leading to:

T 1 T 1 XY = I + uv A− uv A− = I . − In the same way, it is veriﬁed that

1 T 1 1 A− uv A− T YX = A− T 1 (A + uv )= I . − 1 + v A− u