L. Vandenberghe ECE133A (Spring 2021) 5. Orthogonal matrices
matrices with orthonormal columns • orthogonal matrices • tall matrices with orthonormal columns • complex matrices with orthonormal columns •
5.1 Orthonormal vectors
a collection of real <-vectors 01, 02,..., 0= is orthonormal if
the vectors have unit norm: 08 = 1 • k k ) they are mutually orthogonal: 0 0 9 = 0 if 8 ≠ 9 • 8
Example
0 1 1 1 1 0 , 1 , 1 √ √ − 1 2 0 2 0 −
Orthogonal matrices 5.2 Matrix with orthonormal columns
< = R has orthonormal columns if its Gram matrix is the identity matrix: ∈ × ) ) = 0 0 0= 0 0 0= 1 2 ··· 1 2 ··· 0) 0 0) 0 0) 0 1 2 = )1 )1 ··· )1 0 01 0 02 0 0= = 2. 2. ···. . 2. . . . . ) ) ) 0= 0 0= 0 0= 0= 1 2 ··· 1 0 0 ··· 0 1 0 = . . ···. . . . . . . 0 0 1 ···
there is no standard short name for “matrix with orthonormal columns”
Orthogonal matrices 5.3 Matrix-vector product
< = if R has orthonormal columns, then the linear function 5 G = G ∈ × ( )
preserves inner products: • G ) H = G) ) H = G) H ( ) ( )
preserves norms: • 1 2 ) / ) 1 2 G = G G = G G / = G k k ( ) ( ) ( ) k k
preserves distances: G H = G H • k − k k − k
preserves angles: • G ) H G) H ∠ G, H = arccos ( ) ( ) = arccos = ∠ G,H ( ) G H G H ( ) k kk k k kk k
Orthogonal matrices 5.4 Left-invertibility
< = if R has orthonormal columns, then ∈ × ) is left-invertible with left inverse : by definition • ) =
has linearly independent columns (from page 4.24 or page 5.2): • ) G = 0 = G = G = 0 ⇒
is tall or square: < = (see page 4.13) • ≥
Orthogonal matrices 5.5 Outline
matrices with orthonormal columns • orthogonal matrices • tall matrices with orthonormal columns • complex matrices with orthonormal columns • Orthogonal matrix
Orthogonal matrix a square real matrix with orthonormal columns is called orthogonal
Nonsingularity (from equivalences on page 4.14): if is orthogonal, then
) is invertible, with inverse : • ) = = ) = is square ⇒
) is also an orthogonal matrix • rows of are orthonormal (have norm one and are mutually orthogonal) •
< = ) Note: if R has orthonormal columns and < > =, then ≠ ∈ ×
Orthogonal matrices 5.6 Permutation matrix
let c = c , c , . . . , c= be a permutation (reordering) of 1, 2, . . . , = • ( 1 2 ) ( ) we associate with c the = = permutation matrix • ×
8c8 = 1, 8 9 = 0 if 9 ≠ c8
G is a permutation of the elements of G: G = Gc ,Gc ,...,Gc • ( 1 2 =) has exactly one element equal to 1 in each row and each column •
Orthogonality: permutation matrices are orthogonal ) = because has exactly one element equal to one in each row • = ) X 1 8 = 9 8 9 = :8 : 9 = 0 otherwise ( ) :=1
) = 1 is the inverse permutation matrix • −
Orthogonal matrices 5.7 Example
permutation on 1, 2, 3, 4 • { } c , c , c , c = 2, 4, 1, 3 ( 1 2 3 4) ( )
corresponding permutation matrix and its inverse • 0 1 0 0 0 0 1 0 0 0 0 1 1 ) 1 0 0 0 = , − = = 1 0 0 0 0 0 0 1 0 0 1 0 0 1 0 0
) is permutation matrix associated with the permutation • c˜ , c˜ , c˜ , c˜ = 3, 1, 4, 2 ( 1 2 3 4) ( )
Orthogonal matrices 5.8 Plane rotation
Rotation in a plane Ax cos \ sin \ = − sin \ cos \ θ x
= Rotation in a coordinate plane in R : for example,
cos \ 0 sin \ − = 0 1 0 sin \ 0 cos \ describes a rotation in the G ,G plane in R3 ( 1 3)
Orthogonal matrices 5.9 Reflector
Reflector: a matrix of the form
) = 200 − with 0 a unit-norm vector ( 0 = 1) k k
Properties
a reflector matrix is symmetric • a reflector matrix is orthogonal • ) ) ) ) ) ) = 200 200 = 400 400 00 = ( − )( − ) − +
Orthogonal matrices 5.10 Geometrical interpretation of reflector
x
0 H y = I aaT x ( − )
line through a and origin z = Ax = I 2aaT x ( − )
) = D 0 D = 0 is the (hyper-)plane of vectors orthogonal to 0 • { | } if 0 = 1, the projection of G on is given by • k k H = G 0)G 0 = G 0 0)G = 00) G − ( ) − ( ) ( − ) (see next page) reflection of G through the hyperplane is given by product with reflector: • ) I = H H G = 200 G + ( − ) ( − )
Orthogonal matrices 5.11 Exercise
) suppose 0 = 1; show that the projection of G on = D 0 D = 0 is k k { | } H = G 0)G 0 − ( )
we verify that H : • ∈ ) ) ) ) ) ) ) ) 0 H = 0 G 0 0 G = 0 G 0 0 0 G = 0 G 0 G = 0 ( − ( )) − ( )( ) −
now consider any I with I ≠ H and show that G I > G H : • ∈ k − k k − k G I 2 = G H H I 2 k − k k − + − k ) = G H 2 2 G H H I H I 2 k − k + ( )− )) ( − ) + k − k = G H 2 2 0 G 0 H I H I 2 k − k + ( ) ( − ) + k − k) ) = G H 2 H I 2 (because 0 H = 0 I = 0) k − k + k − k > G H 2 k − k
Orthogonal matrices 5.12 Product of orthogonal matrices
if 1,..., : are orthogonal matrices and of equal size, then the product
= : 1 2 ··· is orthogonal:
) ) = 12 : 12 : ( ) ···) ) ) ( ··· ) = : : ··· 2 1 1 2 ··· =
Orthogonal matrices 5.13 Linear equation with orthogonal matrix linear equation with orthogonal coefficient matrix of size = = × G = 1 solution is 1 ) G = − 1 = 1
can be computed in 2=2 flops by matrix-vector multiplication • cost is less than order =2 if has special properties; for example, • permutation matrix: 0 flops reflector (given 0): order = flops plane rotation: order 1 flops
Orthogonal matrices 5.14 Outline
matrices with orthonormal columns • orthogonal matrices • tall matrices with orthonormal columns • complex matrices with orthonormal columns • Tall matrix with orthonormal columns
< = suppose R is tall (< > =) and has orthonormal columns ∈ × ) is a left inverse of : • ) =
has no right inverse; in particular • ) ≠
) on the next pages, we give a geometric interpretation to the matrix
Orthogonal matrices 5.15 Range
the span of a collection of vectors is the set of all their linear combinations: • = span 0 , 0 , . . . , 0= = G 0 G 0 G=0= G R ( 1 2 ) { 1 1 + 2 2 + · · · + | ∈ }
< = the range of a matrix R is the span of its column vectors: • ∈ × = range = G G R ( ) { | ∈ }
Example 1 0 G 1 range 1 2 = G 2G G ,G R ( ) 1 + 2 | 1 2 ∈ 0 1 G − − 2
Orthogonal matrices 5.16 Projection on range of matrix with orthonormal columns
< = suppose R has orthonormal columns; we show that the vector ∈ × ) 1 is the orthogonal projection of an <-vector 1 on range ( ) b
AAT b
range A ( )
) Gˆ = 1 satisfies Gˆ 1 < G 1 for all G ≠ Gˆ • k − k k − k this extends the result on page 2.12 (where = 1 0 0) • ( /k k)
Orthogonal matrices 5.17 Proof the squared distance of 1 to an arbitrary point G in range is ( ) ) G 1 2 = G Gˆ Gˆ 1 2 (where Gˆ = 1) k − k k ( − ) + − k ) ) = G Gˆ 2 Gˆ 1 2 2 G Gˆ Gˆ 1 k ( − )k + k − k + ( − ) ( − ) = G Gˆ 2 Gˆ 1 2 k ( − )k + k − k = G Gˆ 2 Gˆ 1 2 k − k + k − k Gˆ 1 2 ≥ k − k with equality only if G = Gˆ
) ) line 3 follows because Gˆ 1 = Gˆ 1 = 0 • ( − ) − ) line 4 follows from = •
Orthogonal matrices 5.18 Outline
matrices with orthonormal columns • orthogonal matrices • tall matrices with orthonormal columns • complex matrices with orthonormal columns • Gram matrix
< = C has orthonormal columns if its Gram matrix is the identity matrix: ∈ × = 0 0 0= 0 0 0= 1 2 ··· 1 2 ··· 0 0 0 0 0 0 1 2 = 1 1 ··· 1 0 01 0 02 0 0= = 2. 2. ··· 2. . . . 0= 0 0= 0 0= 0= 1 2 ··· 1 0 0 ··· 0 1 0 = . . ···. . . . . . . 0 0 1 ···
columns have unit norm: 08 2 = 0 08 = 1 • k k 8 columns are mutually orthogonal: 0 0 9 = 0 for 8 ≠ 9 • 8
Orthogonal matrices 5.19 Unitary matrix
Unitary matrix a square complex matrix with orthonormal columns is called unitary
Inverse
= = = is square ⇒
a unitary matrix is nonsingular with inverse • if is unitary, then is unitary •
Orthogonal matrices 5.20 Discrete Fourier transform matrix
c = recall definition from page 3.37 (with l = 42 j and j = √ 1) / −
1 1 1 1 1 2 ··· = 1 1 l− l− l−( − ) 2 4 ··· 2 = 1 , = 1 l− l− l− ( − ) . . . ··· . . . . . = = = = 1 l 1 l 2 1 l 1 1 −( − ) − ( − ) ··· −( − )( − ) the matrix 1 √= , is unitary (proof on next page): ( / )
1 1 , , = ,, = = =
inverse of , is , 1 = 1 = , • − ( / ) inverse discrete Fourier transform of =-vector G is , 1G = 1 = , G • − ( / )
Orthogonal matrices 5.21 Gram matrix of DFT matrix
we show that , , = =
conjugate transpose of , is • 1 1 1 1 2 ··· = 1 1 l l l − 2 4 ··· 2 = 1 , = 1 l l l ( − ) . . . ··· . . . . . = = = = 1 l 1 l2 1 l 1 1 − ( − ) ··· ( − )( − )
8, 9 element of Gram matrix is • 8 9 2 8 9 = 1 8 9 , , 8 9 = 1 l − l ( − ) l( − )( − ) ( ) + + + · · · + = 8 9 l ( − ) 1 , , 88 = =, , , 8 9 = = 8 ≠ 9 l8 9 − 0 if ( ) ( ) − 1 = − (last step follows from l = 1)
Orthogonal matrices 5.22