BlockMatrixComputations and the Singular Value Decomposition
ATaleofTwoIdeas
Charles F. Van Loan Department of Computer Science Cornell University
Supported in part by the NSF contract CCR-9901988. Block Matrices
A block matrix is a matrix with matrix entries, e.g.,
×××× ×××× A11 A12 3 2 A = = A IR ×××× ij × A21 A22 ∈ ×××× ×××× ××××
Operations are pretty much “business as usual”, e.g.
T T T T T A11 A12 A11 A12 A11 A21 A11 A12 A11A11 + A21A21 etc = = T T A21 A22 A21 A22 A A A21 A22 etc etc 12 22 E.g., Strassen Multiplication
C11 C12 A11 A12 B11 B12 = C21 C22 A21 A22 B21 B22
P1 =(A11 + A22)(B11 + B22) P2 =(A21 + A22)B11 P3 = A11(B12 B22) − P4 = A22(B21 B11) − P5 =(A11 + A12)B22 P6 =(A21 A11)(B11 + B12) − P7 =(A12 A22)(B21 + B22) − C11 = P1 + P4 P5 + P7 − C12 = P3 + P5 C21 = P2 + P4 C22 = P1 + P3 P2 + P6 − Singular Value Decomposition (SVD)
m n m m n n If A IR × then there exist orthogonal U IR × and V IR × so ∈ ∈ ∈
σ1 0 T U AV = Σ =diag(σ1,...,σn)= 0 σ2 00 where σ1 σ2 σn are the singular values. ≥ ≥ ···≥ Singular Value Decomposition (SVD)
m n m m n n If A IR × then there exist orthogonal U IR × and V IR × so ∈ ∈ ∈
σ1 0 T U AV = Σ =diag(σ1,...,σn)= 0 σ2 00 where σ1 σ2 σn are the singular values. ≥ ≥ ···≥
Fact 1. The columns of V = v1 vn and U = u1 un are the right and left singular vectors and··· they are related: ···
Avj = σjuj j =1:n T A uj = σjvj Singular Value Decomposition (SVD)
m n m m n n If A IR × then there exist orthogonal U IR × and V IR × so ∈ ∈ ∈
σ1 0 T U AV = Σ =diag(σ1,...,σn)= 0 σ2 00 where σ1 σ2 σn are the singular values. ≥ ≥ ···≥
Fact 2. The SVD of A is related to the eigen-decompositions of AT A and AAT :
T T 2 2 V (A A)V =diag(σ1,...,σn)
T T 2 2 U (AA )U =diag(σ1,...,σn, 0,...,0) m n − Singular Value Decomposition (SVD)
m n m m n n If A IR × then there exist orthogonal U IR × and V IR × so ∈ ∈ ∈
σ1 0 T U AV = Σ =diag(σ1,...,σn)= 0 σ2 00 where σ1 σ2 σn are the singular values. ≥ ≥ ···≥
Fact 3. The smallest singular value is the distance from A to the set of rank deficient matrices:
σmin = min A B F rank(B)