AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 2: Orthogonal Vectors and Matrices; Vector Norms

Xiangmin Jiao

SUNY Stony Brook

Xiangmin Jiao Numerical Analysis I 1 / 11 Outline

1 Orthogonal Vectors and Matrices

2 Vector Norms

Xiangmin Jiao Numerical Analysis I 2 / 11 Euclidean length√ of u is the square root of the inner product of u with itself, i.e., u∗u

Inner product of two unit vectors u and v is the cosine of the angle α u∗v between u and v, i.e., cos α = kukkv k Inner product is bilinear, in the sense that it is linear in each vertex separately:

∗ ∗ ∗ (u1 + u2) v = u1v + u2v ∗ ∗ ∗ u (v 1 + v 2) = u v 1 + u v 2 (αu)∗(βv) =αβ ¯ u∗v

Inner Product m Inner product (dot product) of two column vectors u, v ∈ C is ∗ Pm u v = i=1 u¯i vi In contrast, outer product of u and v is uv ∗ Note that cross product is diﬀerent

Xiangmin Jiao Numerical Analysis I 3 / 11 Inner product of two unit vectors u and v is the cosine of the angle α u∗v between u and v, i.e., cos α = kukkv k Inner product is bilinear, in the sense that it is linear in each vertex separately:

∗ ∗ ∗ (u1 + u2) v = u1v + u2v ∗ ∗ ∗ u (v 1 + v 2) = u v 1 + u v 2 (αu)∗(βv) =αβ ¯ u∗v

Inner Product m Inner product (dot product) of two column vectors u, v ∈ C is ∗ Pm u v = i=1 u¯i vi In contrast, outer product of u and v is uv ∗ Note that cross product is diﬀerent

Euclidean length√ of u is the square root of the inner product of u with itself, i.e., u∗u

Xiangmin Jiao Numerical Analysis I 3 / 11 Inner product is bilinear, in the sense that it is linear in each vertex separately:

∗ ∗ ∗ (u1 + u2) v = u1v + u2v ∗ ∗ ∗ u (v 1 + v 2) = u v 1 + u v 2 (αu)∗(βv) =αβ ¯ u∗v

Inner Product m Inner product (dot product) of two column vectors u, v ∈ C is ∗ Pm u v = i=1 u¯i vi In contrast, outer product of u and v is uv ∗ Note that cross product is diﬀerent

Euclidean length√ of u is the square root of the inner product of u with itself, i.e., u∗u

Inner product of two unit vectors u and v is the cosine of the angle α u∗v between u and v, i.e., cos α = kukkv k

Xiangmin Jiao Numerical Analysis I 3 / 11 Inner Product m Inner product (dot product) of two column vectors u, v ∈ C is ∗ Pm u v = i=1 u¯i vi In contrast, outer product of u and v is uv ∗ Note that cross product is diﬀerent

Euclidean length√ of u is the square root of the inner product of u with itself, i.e., u∗u

∗ ∗ ∗ (u1 + u2) v = u1v + u2v ∗ ∗ ∗ u (v 1 + v 2) = u v 1 + u v 2 (αu)∗(βv) =αβ ¯ u∗v

Xiangmin Jiao Numerical Analysis I 3 / 11 Deﬁnition Two sets of vectors X and Y are orthogonal if every x ∈ X is orthogonal to every y ∈ Y .

Deﬁnition A set of nonzero vectors S is orthogonal if they are pairwise orthogonal. They are orthonormal if it is orthogonal and in addition each vector has unit Euclidean length.

Orthogonal Vectors

Deﬁnition A pair of vectors are orthogonal if x∗y = 0.

In other words, the angle between them is 90 degrees

Xiangmin Jiao Numerical Analysis I 4 / 11 Deﬁnition A set of nonzero vectors S is orthogonal if they are pairwise orthogonal. They are orthonormal if it is orthogonal and in addition each vector has unit Euclidean length.

Orthogonal Vectors

Deﬁnition A pair of vectors are orthogonal if x∗y = 0.

In other words, the angle between them is 90 degrees Deﬁnition Two sets of vectors X and Y are orthogonal if every x ∈ X is orthogonal to every y ∈ Y .

Xiangmin Jiao Numerical Analysis I 4 / 11 Orthogonal Vectors

Deﬁnition A pair of vectors are orthogonal if x∗y = 0.

In other words, the angle between them is 90 degrees Deﬁnition Two sets of vectors X and Y are orthogonal if every x ∈ X is orthogonal to every y ∈ Y .

Deﬁnition A set of nonzero vectors S is orthogonal if they are pairwise orthogonal. They are orthonormal if it is orthogonal and in addition each vector has unit Euclidean length.

Xiangmin Jiao Numerical Analysis I 4 / 11 Proof. Prove by contradiction. If a vector can be expressed as linear combination of the other vectors in the set, then it is orthogonal to itself.

Question: If the column vectors of an m × n matrix A are orthogonal, what is the rank of A? Answer: n = min{m, n}. In other words, A has full rank.

Orthogonal Vectors

Theorem The vectors in an orthogonal set S are linearly independent.

Xiangmin Jiao Numerical Analysis I 5 / 11 Answer: n = min{m, n}. In other words, A has full rank.

Orthogonal Vectors

Theorem The vectors in an orthogonal set S are linearly independent.

Proof. Prove by contradiction. If a vector can be expressed as linear combination of the other vectors in the set, then it is orthogonal to itself.

Question: If the column vectors of an m × n matrix A are orthogonal, what is the rank of A?

Xiangmin Jiao Numerical Analysis I 5 / 11 Orthogonal Vectors

Theorem The vectors in an orthogonal set S are linearly independent.

Proof. Prove by contradiction. If a vector can be expressed as linear combination of the other vectors in the set, then it is orthogonal to itself.

Question: If the column vectors of an m × n matrix A are orthogonal, what is the rank of A? Answer: n = min{m, n}. In other words, A has full rank.

Xiangmin Jiao Numerical Analysis I 5 / 11 Pm ∗ Another way to express the condition is v = i=1(qi qi )v ∗ qi qi is an orthogonal projection matrix. Note that it is NOT an orthogonal matrix.

More generally, given an orthonormal set {q1, q2,..., qn} with n ≤ m, we have

n n X ∗ X ∗ ∗ v = r + (qi v)qi = r + (qi qi )v and r qi = 0, 1 ≤ i ≤ n i=1 i=1

Let Q be composed of column vectors {q1, q2,..., qn}. ∗ Pn ∗ QQ = i=1(qi qi ) is an orthogonal projection matrix.

Components of Vector

m Given an orthonormal set {q1, q2,..., qm} forming a basis of C , vector v can be decomposed into orthogonal components as Pm ∗ v = i=1(qi v)qi

Xiangmin Jiao Numerical Analysis I 6 / 11 More generally, given an orthonormal set {q1, q2,..., qn} with n ≤ m, we have

n n X ∗ X ∗ ∗ v = r + (qi v)qi = r + (qi qi )v and r qi = 0, 1 ≤ i ≤ n i=1 i=1

Let Q be composed of column vectors {q1, q2,..., qn}. ∗ Pn ∗ QQ = i=1(qi qi ) is an orthogonal projection matrix.

Components of Vector

m Given an orthonormal set {q1, q2,..., qm} forming a basis of C , vector v can be decomposed into orthogonal components as Pm ∗ v = i=1(qi v)qi

Pm ∗ Another way to express the condition is v = i=1(qi qi )v ∗ qi qi is an orthogonal projection matrix. Note that it is NOT an orthogonal matrix.

Xiangmin Jiao Numerical Analysis I 6 / 11 Components of Vector

m Given an orthonormal set {q1, q2,..., qm} forming a basis of C , vector v can be decomposed into orthogonal components as Pm ∗ v = i=1(qi v)qi

Pm ∗ Another way to express the condition is v = i=1(qi qi )v ∗ qi qi is an orthogonal projection matrix. Note that it is NOT an orthogonal matrix.

More generally, given an orthonormal set {q1, q2,..., qn} with n ≤ m, we have

n n X ∗ X ∗ ∗ v = r + (qi v)qi = r + (qi qi )v and r qi = 0, 1 ≤ i ≤ n i=1 i=1

Let Q be composed of column vectors {q1, q2,..., qn}. ∗ Pn ∗ QQ = i=1(qi qi ) is an orthogonal projection matrix.

Xiangmin Jiao Numerical Analysis I 6 / 11 Question: What is the geometric meaning of multiplication by a unitary matrix? Answer: It preserves angles and Euclidean length. In the real case, multiplication by an orthogonal matrix Q is a rotation (if det(Q) = 1) or reﬂection (if det(Q) = −1).

Unitary Matrices

Deﬁnition A matrix is unitary if Q∗ = Q−1, i.e., if Q∗Q = QQ∗ = I .

In the real case, we say the matrix is orthogonal. Its column vectors are orthonormal. ∗ In other words, qi qj = δij , the Kronecker delta.

Xiangmin Jiao Numerical Analysis I 7 / 11 Answer: It preserves angles and Euclidean length. In the real case, multiplication by an orthogonal matrix Q is a rotation (if det(Q) = 1) or reﬂection (if det(Q) = −1).

Unitary Matrices

Deﬁnition A matrix is unitary if Q∗ = Q−1, i.e., if Q∗Q = QQ∗ = I .

Xiangmin Jiao Numerical Analysis I 7 / 11 Unitary Matrices

Deﬁnition A matrix is unitary if Q∗ = Q−1, i.e., if Q∗Q = QQ∗ = I .

In the real case, we say the matrix is orthogonal. Its column vectors are orthonormal. ∗ In other words, qi qj = δij , the Kronecker delta. Question: What is the geometric meaning of multiplication by a unitary matrix? Answer: It preserves angles and Euclidean length. In the real case, multiplication by an orthogonal matrix Q is a rotation (if det(Q) = 1) or reﬂection (if det(Q) = −1).

Xiangmin Jiao Numerical Analysis I 7 / 11 Outline

1 Orthogonal Vectors and Matrices

2 Vector Norms

Xiangmin Jiao Numerical Analysis I 8 / 11 Deﬁnition m A norm is a function k · k : C → R that assigns a real-valued length to each vector. It must satisfy the following conditions:

(1)kxk ≥ 0, and kxk = 0 only if x = 0, (2)kx + yk ≤ kxk + kyk, (3)kαxk = |α|kxk.

pPm 2 An example is Euclidean length (i.e, kxk = i=1 |xi | )

Deﬁnition of Norms

Norm captures “size” of vector or “distance” between vectors There are many diﬀerent measures for “sizes” but a norm must satisfy some requirements:

Xiangmin Jiao Numerical Analysis I 9 / 11 pPm 2 An example is Euclidean length (i.e, kxk = i=1 |xi | )

Deﬁnition of Norms

Norm captures “size” of vector or “distance” between vectors There are many diﬀerent measures for “sizes” but a norm must satisfy some requirements:

Deﬁnition m A norm is a function k · k : C → R that assigns a real-valued length to each vector. It must satisfy the following conditions:

(1)kxk ≥ 0, and kxk = 0 only if x = 0, (2)kx + yk ≤ kxk + kyk, (3)kαxk = |α|kxk.

Xiangmin Jiao Numerical Analysis I 9 / 11 Deﬁnition of Norms

Norm captures “size” of vector or “distance” between vectors There are many diﬀerent measures for “sizes” but a norm must satisfy some requirements:

Deﬁnition m A norm is a function k · k : C → R that assigns a real-valued length to each vector. It must satisfy the following conditions:

(1)kxk ≥ 0, and kxk = 0 only if x = 0, (2)kx + yk ≤ kxk + kyk, (3)kαxk = |α|kxk.

pPm 2 An example is Euclidean length (i.e, kxk = i=1 |xi | )

Xiangmin Jiao Numerical Analysis I 9 / 11 Euclidean norm is 2-norm kxk2 (i.e., p = 2)

Pm 1-norm: kxk1 = i=1 |xi |

∞-norm: kxk∞ . What is its value?

I Answer: kxk∞ = max1≤i≤m |xi | Why we require p ≥ 1? What happens if 0 ≤ p < 1?

p-norms

Euclidean length is a special case of p-norms, deﬁned as

m !1/p X p kxkp = |xi | i=1 for 1 ≤ p ≤ ∞

Xiangmin Jiao Numerical Analysis I 10 / 11 Pm 1-norm: kxk1 = i=1 |xi |

∞-norm: kxk∞ . What is its value?

I Answer: kxk∞ = max1≤i≤m |xi | Why we require p ≥ 1? What happens if 0 ≤ p < 1?

p-norms

Euclidean length is a special case of p-norms, deﬁned as

m !1/p X p kxkp = |xi | i=1 for 1 ≤ p ≤ ∞

Euclidean norm is 2-norm kxk2 (i.e., p = 2)

Xiangmin Jiao Numerical Analysis I 10 / 11 ∞-norm: kxk∞ . What is its value?

I Answer: kxk∞ = max1≤i≤m |xi | Why we require p ≥ 1? What happens if 0 ≤ p < 1?

p-norms

Euclidean length is a special case of p-norms, deﬁned as

m !1/p X p kxkp = |xi | i=1 for 1 ≤ p ≤ ∞

Euclidean norm is 2-norm kxk2 (i.e., p = 2)

Pm 1-norm: kxk1 = i=1 |xi |

Xiangmin Jiao Numerical Analysis I 10 / 11 I Answer: kxk∞ = max1≤i≤m |xi | Why we require p ≥ 1? What happens if 0 ≤ p < 1?

p-norms

Euclidean length is a special case of p-norms, deﬁned as

m !1/p X p kxkp = |xi | i=1 for 1 ≤ p ≤ ∞

Euclidean norm is 2-norm kxk2 (i.e., p = 2)

Pm 1-norm: kxk1 = i=1 |xi |

∞-norm: kxk∞ . What is its value?

Xiangmin Jiao Numerical Analysis I 10 / 11 Why we require p ≥ 1? What happens if 0 ≤ p < 1?

p-norms

Euclidean length is a special case of p-norms, deﬁned as

m !1/p X p kxkp = |xi | i=1 for 1 ≤ p ≤ ∞

Euclidean norm is 2-norm kxk2 (i.e., p = 2)

Pm 1-norm: kxk1 = i=1 |xi |

∞-norm: kxk∞ . What is its value?

I Answer: kxk∞ = max1≤i≤m |xi |

Xiangmin Jiao Numerical Analysis I 10 / 11 p-norms

Euclidean length is a special case of p-norms, deﬁned as

m !1/p X p kxkp = |xi | i=1 for 1 ≤ p ≤ ∞

Euclidean norm is 2-norm kxk2 (i.e., p = 2)

Pm 1-norm: kxk1 = i=1 |xi |

∞-norm: kxk∞ . What is its value?

I Answer: kxk∞ = max1≤i≤m |xi | Why we require p ≥ 1? What happens if 0 ≤ p < 1?

Xiangmin Jiao Numerical Analysis I 10 / 11 Can we further generalize it to allow W being arbitrary matrix? No. But we can allow W to be arbitrary nonsingular matrix.

Weighted p-norms

A generalization of p-norm is weighted p-norm, which assigns diﬀerent weights (priorities) to diﬀerent components.

I It is anisotropic instead of isotropic

Algebraically, kxkW = kW xk, where W is diagonal matrix with ith diagonal entry wi 6= 0 being weight for ith component In other words, m !1/p X p kxkW = |wi xi | i=1

What happens if we allow wi = 0?

Xiangmin Jiao Numerical Analysis I 11 / 11 Weighted p-norms

A generalization of p-norm is weighted p-norm, which assigns diﬀerent weights (priorities) to diﬀerent components.

I It is anisotropic instead of isotropic

Algebraically, kxkW = kW xk, where W is diagonal matrix with ith diagonal entry wi 6= 0 being weight for ith component In other words, m !1/p X p kxkW = |wi xi | i=1

What happens if we allow wi = 0?

Can we further generalize it to allow W being arbitrary matrix? No. But we can allow W to be arbitrary nonsingular matrix.

Xiangmin Jiao Numerical Analysis I 11 / 11