AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 2: Orthogonal Vectors and Matrices; Vector Norms Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 11 Outline 1 Orthogonal Vectors and Matrices 2 Vector Norms Xiangmin Jiao Numerical Analysis I 2 / 11 Euclidean lengthp of u is the square root of the inner product of u with itself, i.e., u∗u Inner product of two unit vectors u and v is the cosine of the angle α u∗v between u and v, i.e., cos α = kukkv k Inner product is bilinear, in the sense that it is linear in each vertex separately: ∗ ∗ ∗ (u1 + u2) v = u1v + u2v ∗ ∗ ∗ u (v 1 + v 2) = u v 1 + u v 2 (αu)∗(βv) =αβ ¯ u∗v Inner Product m Inner product (dot product) of two column vectors u; v 2 C is ∗ Pm u v = i=1 u¯i vi In contrast, outer product of u and v is uv ∗ Note that cross product is different Xiangmin Jiao Numerical Analysis I 3 / 11 Inner product of two unit vectors u and v is the cosine of the angle α u∗v between u and v, i.e., cos α = kukkv k Inner product is bilinear, in the sense that it is linear in each vertex separately: ∗ ∗ ∗ (u1 + u2) v = u1v + u2v ∗ ∗ ∗ u (v 1 + v 2) = u v 1 + u v 2 (αu)∗(βv) =αβ ¯ u∗v Inner Product m Inner product (dot product) of two column vectors u; v 2 C is ∗ Pm u v = i=1 u¯i vi In contrast, outer product of u and v is uv ∗ Note that cross product is different Euclidean lengthp of u is the square root of the inner product of u with itself, i.e., u∗u Xiangmin Jiao Numerical Analysis I 3 / 11 Inner product is bilinear, in the sense that it is linear in each vertex separately: ∗ ∗ ∗ (u1 + u2) v = u1v + u2v ∗ ∗ ∗ u (v 1 + v 2) = u v 1 + u v 2 (αu)∗(βv) =αβ ¯ u∗v Inner Product m Inner product (dot product) of two column vectors u; v 2 C is ∗ Pm u v = i=1 u¯i vi In contrast, outer product of u and v is uv ∗ Note that cross product is different Euclidean lengthp of u is the square root of the inner product of u with itself, i.e., u∗u Inner product of two unit vectors u and v is the cosine of the angle α u∗v between u and v, i.e., cos α = kukkv k Xiangmin Jiao Numerical Analysis I 3 / 11 Inner Product m Inner product (dot product) of two column vectors u; v 2 C is ∗ Pm u v = i=1 u¯i vi In contrast, outer product of u and v is uv ∗ Note that cross product is different Euclidean lengthp of u is the square root of the inner product of u with itself, i.e., u∗u Inner product of two unit vectors u and v is the cosine of the angle α u∗v between u and v, i.e., cos α = kukkv k Inner product is bilinear, in the sense that it is linear in each vertex separately: ∗ ∗ ∗ (u1 + u2) v = u1v + u2v ∗ ∗ ∗ u (v 1 + v 2) = u v 1 + u v 2 (αu)∗(βv) =αβ ¯ u∗v Xiangmin Jiao Numerical Analysis I 3 / 11 Definition Two sets of vectors X and Y are orthogonal if every x 2 X is orthogonal to every y 2 Y . Definition A set of nonzero vectors S is orthogonal if they are pairwise orthogonal. They are orthonormal if it is orthogonal and in addition each vector has unit Euclidean length. Orthogonal Vectors Definition A pair of vectors are orthogonal if x∗y = 0. In other words, the angle between them is 90 degrees Xiangmin Jiao Numerical Analysis I 4 / 11 Definition A set of nonzero vectors S is orthogonal if they are pairwise orthogonal. They are orthonormal if it is orthogonal and in addition each vector has unit Euclidean length. Orthogonal Vectors Definition A pair of vectors are orthogonal if x∗y = 0. In other words, the angle between them is 90 degrees Definition Two sets of vectors X and Y are orthogonal if every x 2 X is orthogonal to every y 2 Y . Xiangmin Jiao Numerical Analysis I 4 / 11 Orthogonal Vectors Definition A pair of vectors are orthogonal if x∗y = 0. In other words, the angle between them is 90 degrees Definition Two sets of vectors X and Y are orthogonal if every x 2 X is orthogonal to every y 2 Y . Definition A set of nonzero vectors S is orthogonal if they are pairwise orthogonal. They are orthonormal if it is orthogonal and in addition each vector has unit Euclidean length. Xiangmin Jiao Numerical Analysis I 4 / 11 Proof. Prove by contradiction. If a vector can be expressed as linear combination of the other vectors in the set, then it is orthogonal to itself. Question: If the column vectors of an m × n matrix A are orthogonal, what is the rank of A? Answer: n = minfm; ng. In other words, A has full rank. Orthogonal Vectors Theorem The vectors in an orthogonal set S are linearly independent. Xiangmin Jiao Numerical Analysis I 5 / 11 Answer: n = minfm; ng. In other words, A has full rank. Orthogonal Vectors Theorem The vectors in an orthogonal set S are linearly independent. Proof. Prove by contradiction. If a vector can be expressed as linear combination of the other vectors in the set, then it is orthogonal to itself. Question: If the column vectors of an m × n matrix A are orthogonal, what is the rank of A? Xiangmin Jiao Numerical Analysis I 5 / 11 Orthogonal Vectors Theorem The vectors in an orthogonal set S are linearly independent. Proof. Prove by contradiction. If a vector can be expressed as linear combination of the other vectors in the set, then it is orthogonal to itself. Question: If the column vectors of an m × n matrix A are orthogonal, what is the rank of A? Answer: n = minfm; ng. In other words, A has full rank. Xiangmin Jiao Numerical Analysis I 5 / 11 Pm ∗ Another way to express the condition is v = i=1(qi qi )v ∗ qi qi is an orthogonal projection matrix. Note that it is NOT an orthogonal matrix. More generally, given an orthonormal set fq1; q2;:::; qng with n ≤ m, we have n n X ∗ X ∗ ∗ v = r + (qi v)qi = r + (qi qi )v and r qi = 0; 1 ≤ i ≤ n i=1 i=1 Let Q be composed of column vectors fq1; q2;:::; qng. ∗ Pn ∗ QQ = i=1(qi qi ) is an orthogonal projection matrix. Components of Vector m Given an orthonormal set fq1; q2;:::; qmg forming a basis of C , vector v can be decomposed into orthogonal components as Pm ∗ v = i=1(qi v)qi Xiangmin Jiao Numerical Analysis I 6 / 11 More generally, given an orthonormal set fq1; q2;:::; qng with n ≤ m, we have n n X ∗ X ∗ ∗ v = r + (qi v)qi = r + (qi qi )v and r qi = 0; 1 ≤ i ≤ n i=1 i=1 Let Q be composed of column vectors fq1; q2;:::; qng. ∗ Pn ∗ QQ = i=1(qi qi ) is an orthogonal projection matrix. Components of Vector m Given an orthonormal set fq1; q2;:::; qmg forming a basis of C , vector v can be decomposed into orthogonal components as Pm ∗ v = i=1(qi v)qi Pm ∗ Another way to express the condition is v = i=1(qi qi )v ∗ qi qi is an orthogonal projection matrix. Note that it is NOT an orthogonal matrix. Xiangmin Jiao Numerical Analysis I 6 / 11 Components of Vector m Given an orthonormal set fq1; q2;:::; qmg forming a basis of C , vector v can be decomposed into orthogonal components as Pm ∗ v = i=1(qi v)qi Pm ∗ Another way to express the condition is v = i=1(qi qi )v ∗ qi qi is an orthogonal projection matrix. Note that it is NOT an orthogonal matrix. More generally, given an orthonormal set fq1; q2;:::; qng with n ≤ m, we have n n X ∗ X ∗ ∗ v = r + (qi v)qi = r + (qi qi )v and r qi = 0; 1 ≤ i ≤ n i=1 i=1 Let Q be composed of column vectors fq1; q2;:::; qng. ∗ Pn ∗ QQ = i=1(qi qi ) is an orthogonal projection matrix. Xiangmin Jiao Numerical Analysis I 6 / 11 Question: What is the geometric meaning of multiplication by a unitary matrix? Answer: It preserves angles and Euclidean length. In the real case, multiplication by an orthogonal matrix Q is a rotation (if det(Q) = 1) or reflection (if det(Q) = −1). Unitary Matrices Definition A matrix is unitary if Q∗ = Q−1, i.e., if Q∗Q = QQ∗ = I . In the real case, we say the matrix is orthogonal. Its column vectors are orthonormal. ∗ In other words, qi qj = δij , the Kronecker delta. Xiangmin Jiao Numerical Analysis I 7 / 11 Answer: It preserves angles and Euclidean length. In the real case, multiplication by an orthogonal matrix Q is a rotation (if det(Q) = 1) or reflection (if det(Q) = −1). Unitary Matrices Definition A matrix is unitary if Q∗ = Q−1, i.e., if Q∗Q = QQ∗ = I . In the real case, we say the matrix is orthogonal. Its column vectors are orthonormal. ∗ In other words, qi qj = δij , the Kronecker delta. Question: What is the geometric meaning of multiplication by a unitary matrix? Xiangmin Jiao Numerical Analysis I 7 / 11 Unitary Matrices Definition A matrix is unitary if Q∗ = Q−1, i.e., if Q∗Q = QQ∗ = I .