L. Vandenberghe ECE133A (Spring 2021) 5. Orthogonal matrices matrices with orthonormal columns • orthogonal matrices • tall matrices with orthonormal columns • complex matrices with orthonormal columns • 5.1 Orthonormal vectors a collection of real <-vectors 01, 02,..., 0= is orthonormal if the vectors have unit norm: 08 = 1 • k k ) they are mutually orthogonal: 0 0 9 = 0 if 8 < 9 • 8 Example 2 0 3 2 1 3 2 1 3 6 7 1 6 7 1 6 7 6 0 7 , 6 1 7 , 6 1 7 6 7 p 6 7 p 6 − 7 6 1 7 2 6 0 7 2 6 0 7 4 − 5 4 5 4 5 Orthogonal matrices 5.2 Matrix with orthonormal columns < = R has orthonormal columns if its Gram matrix is the identity matrix: 2 × ) ) = 0 0 0= 0 0 0= 1 2 ··· 1 2 ··· 0) 0 0) 0 0) 0 2 1 2 = 3 6 )1 )1 ··· )1 7 6 0 01 0 02 0 0= 7 = 6 2. 2. ···. 2. 7 6 . 7 6 ) ) ) 7 6 0= 0 0= 0 0= 0= 7 4 1 2 ··· 5 2 1 0 0 3 6 ··· 7 6 0 1 0 7 = 6 . ···. 7 6 . 7 6 7 6 0 0 1 7 4 ··· 5 there is no standard short name for “matrix with orthonormal columns” Orthogonal matrices 5.3 Matrix-vector product < = if R has orthonormal columns, then the linear function 5 G = G 2 × ¹ º preserves inner products: • G ) H = G) ) H = G) H ¹ º ¹ º preserves norms: • 1 2 ) / ) 1 2 G = G G = G G / = G k k ¹ º ¹ º ¹ º k k preserves distances: G H = G H • k − k k − k preserves angles: • G ) H G) H \ G, H = arccos ¹ º ¹ º = arccos = \ G, H ¹ º G H G H ¹ º k kk k k kk k Orthogonal matrices 5.4 Left-invertibility < = if R has orthonormal columns, then 2 × ) is left-invertible with left inverse : by definition • ) = has linearly independent columns (from page 4.24 or page 5.2): • ) G = 0 = G = G = 0 ) is tall or square: < = (see page 4.13) • ≥ Orthogonal matrices 5.5 Outline matrices with orthonormal columns • orthogonal matrices • tall matrices with orthonormal columns • complex matrices with orthonormal columns • Orthogonal matrix Orthogonal matrix a square real matrix with orthonormal columns is called orthogonal Nonsingularity (from equivalences on page 4.14): if is orthogonal, then ) is invertible, with inverse : • ) = = ) = is square ) ) is also an orthogonal matrix • rows of are orthonormal (have norm one and are mutually orthogonal) • < = ) Note: if R has orthonormal columns and < ¡ =, then < 2 × Orthogonal matrices 5.6 Permutation matrix let c = c , c , . , c= be a permutation (reordering) of 1, 2, . , = • ¹ 1 2 º ¹ º we associate with c the = = permutation matrix • × 8c8 = 1, 8 9 = 0 if 9 < c8 G is a permutation of the elements of G: G = Gc ,Gc ,...,Gc • ¹ 1 2 =º has exactly one element equal to 1 in each row and each column • Orthogonality: permutation matrices are orthogonal ) = because has exactly one element equal to one in each row • = ) X 1 8 = 9 8 9 = :8 : 9 = 0 otherwise ¹ º :=1 ) = 1 is the inverse permutation matrix • − Orthogonal matrices 5.7 Example permutation on 1, 2, 3, 4 • f g c , c , c , c = 2, 4, 1, 3 ¹ 1 2 3 4º ¹ º corresponding permutation matrix and its inverse • 2 0 1 0 0 3 2 0 0 1 0 3 6 7 6 7 6 0 0 0 1 7 1 ) 6 1 0 0 0 7 = 6 7 , − = = 6 7 6 1 0 0 0 7 6 0 0 0 1 7 6 7 6 7 6 0 0 1 0 7 6 0 1 0 0 7 4 5 4 5 ) is permutation matrix associated with the permutation • c˜ , c˜ , c˜ , c˜ = 3, 1, 4, 2 ¹ 1 2 3 4º ¹ º Orthogonal matrices 5.8 Plane rotation Rotation in a plane Ax cos \ sin \ = − sin \ cos \ θ x = Rotation in a coordinate plane in R : for example, 2 cos \ 0 sin \ 3 6 − 7 = 6 0 1 0 7 6 7 6 sin \ 0 cos \ 7 4 5 describes a rotation in the G ,G plane in R3 ¹ 1 3º Orthogonal matrices 5.9 Reflector Reflector: a matrix of the form ) = 200 − with 0 a unit-norm vector ( 0 = 1) k k Properties a reflector matrix is symmetric • a reflector matrix is orthogonal • ) ) ) ) ) ) = 200 200 = 400 400 00 = ¹ − º¹ − º − ¸ Orthogonal matrices 5.10 Geometrical interpretation of reflector x 0 H y = I aaT x ( − ) line through a and origin z = Ax = I 2aaT x ( − ) ) = D 0 D = 0 is the (hyper-)plane of vectors orthogonal to 0 • f j g if 0 = 1, the projection of G on is given by • k k H = G 0)G 0 = G 0 0)G = 00) G − ¹ º − ¹ º ¹ − º (see next page) reflection of G through the hyperplane is given by product with reflector: • ) I = H H G = 200 G ¸ ¹ − º ¹ − º Orthogonal matrices 5.11 Exercise ) suppose 0 = 1; show that the projection of G on = D 0 D = 0 is k k f j g H = G 0)G 0 − ¹ º we verify that H : • 2 ) ) ) ) ) ) ) ) 0 H = 0 G 0 0 G = 0 G 0 0 0 G = 0 G 0 G = 0 ¹ − ¹ ºº − ¹ º¹ º − now consider any I with I < H and show that G I ¡ G H : • 2 k − k k − k G I 2 = G H H I 2 k − k k − ¸ − k ) = G H 2 2 G H H I H I 2 k − k ¸ ¹ )− º) ¹ − º ¸ k − k = G H 2 2 0 G 0 H I H I 2 k − k ¸ ¹ º ¹ − º ¸ k − k) ) = G H 2 H I 2 (because 0 H = 0 I = 0) k − k ¸ k − k ¡ G H 2 k − k Orthogonal matrices 5.12 Product of orthogonal matrices if 1,..., : are orthogonal matrices and of equal size, then the product = : 1 2 ··· is orthogonal: ) ) = 12 : 12 : ¹ ) ···) ) º ¹ ··· º = : : ··· 2 1 1 2 ··· = Orthogonal matrices 5.13 Linear equation with orthogonal matrix linear equation with orthogonal coefficient matrix of size = = × G = 1 solution is 1 ) G = − 1 = 1 can be computed in 2=2 flops by matrix-vector multiplication • cost is less than order =2 if has special properties; for example, • permutation matrix: 0 flops reflector (given 0): order = flops plane rotation: order 1 flops Orthogonal matrices 5.14 Outline matrices with orthonormal columns • orthogonal matrices • tall matrices with orthonormal columns • complex matrices with orthonormal columns • Tall matrix with orthonormal columns < = suppose R is tall (< ¡ =) and has orthonormal columns 2 × ) is a left inverse of : • ) = has no right inverse; in particular • ) < ) on the next pages, we give a geometric interpretation to the matrix Orthogonal matrices 5.15 Range the span of a collection of vectors is the set of all their linear combinations: • = span 0 , 0 , . , 0= = G 0 G 0 G=0= G R ¹ 1 2 º f 1 1 ¸ 2 2 ¸ · · · ¸ j 2 g < = the range of a matrix R is the span of its column vectors: • 2 × = range = G G R ¹ º f j 2 g Example 2 1 0 3 8 2 G 3 9 6 7 ><> 6 1 7 >=> range 6 1 2 7 = 6 G 2G 7 G ,G R ¹6 7º 6 1 ¸ 2 7 j 1 2 2 6 0 1 7 > 6 G 7 > 4 − 5 : 4 − 2 5 ; Orthogonal matrices 5.16 Projection on range of matrix with orthonormal columns < = suppose R has orthonormal columns; we show that the vector 2 × ) 1 is the orthogonal projection of an <-vector 1 on range ¹ º b AAT b range A ( ) ) Gˆ = 1 satisfies Gˆ 1 < G 1 for all G < Gˆ • k − k k − k this extends the result on page 2.12 (where = 1 0 0) • ¹ /k kº Orthogonal matrices 5.17 Proof the squared distance of 1 to an arbitrary point G in range is ¹ º ) G 1 2 = G Gˆ Gˆ 1 2 (where Gˆ = 1) k − k k ¹ − º ¸ − k ) ) = G Gˆ 2 Gˆ 1 2 2 G Gˆ Gˆ 1 k ¹ − ºk ¸ k − k ¸ ¹ − º ¹ − º = G Gˆ 2 Gˆ 1 2 k ¹ − ºk ¸ k − k = G Gˆ 2 Gˆ 1 2 k − k ¸ k − k Gˆ 1 2 ≥ k − k with equality only if G = Gˆ ) ) line 3 follows because Gˆ 1 = Gˆ 1 = 0 • ¹ − º − ) line 4 follows from = • Orthogonal matrices 5.18 Outline matrices with orthonormal columns • orthogonal matrices • tall matrices with orthonormal columns • complex matrices with orthonormal columns • Gram matrix < = C has orthonormal columns if its Gram matrix is the identity matrix: 2 × = 0 0 0= 0 0 0= 1 2 ··· 1 2 ··· 00 00 00 2 1 2 = 3 6 1 1 ··· 1 7 6 0 01 0 02 0 0= 7 = 6 2. 2. ··· 2. 7 6 . 7 6 7 6 0= 0 0= 0 0= 0= 7 4 1 2 ··· 5 2 1 0 0 3 6 ··· 7 6 0 1 0 7 = 6 . ···. 7 6 . 7 6 7 6 0 0 1 7 4 ··· 5 columns have unit norm: 08 2 = 0 08 = 1 • k k 8 columns are mutually orthogonal: 0 0 9 = 0 for 8 < 9 • 8 Orthogonal matrices 5.19 Unitary matrix Unitary matrix a square complex matrix with orthonormal columns is called unitary Inverse = = = is square ) a unitary matrix is nonsingular with inverse • if is unitary, then is unitary • Orthogonal matrices 5.20 Discrete Fourier transform matrix c = recall definition from page 3.37 (with l = 42 j and j = p 1) / − 2 1 1 1 1 3 6 1 2 ··· = 1 7 6 1 l− l− l−¹ − º 7 6 2 4 ··· 2 = 1 7 , = 6 1 l− l− l− ¹ − º 7 6 .