Technische Universit at Chemnitz Zwickau
Sonderforschungsbereich
Numerische Simulation auf massiv parallelen Rechnern
Brahim Benhammouda
RANK REVEALING
TOP DOWN ULV
FACTORIZATIONS
Preprint SFB
Key words U LV and U RV factorizations Orthogonal factorizations
Rank revealing factorizations Numerical rank Di erential algebraic equations
AMS sub ject classi cations F A F
Author s address
Brahim Benhammouda
Fakult atf ur Mathematik
TU Chemnitz Zwickau
D Chemnitz FRG
e mail brahim mathematik tu chemnitz de
This work has b een supp orted by the Deutsche Forschungsgemeinschaft under
the grant no Me Di erentiell algebraische Gleichungen
Preprint Reihe des Chemnitzer SFB
SFB January
Abstract
Rank revealing U LV and U RV factorizations are useful to ols to
determine the rank and to compute bases for null spaces of a ma
trix However in the practical U LV resp U RV factorization each
left resp right null vector is recomputed from its corresp onding
right resp left null vector via triangular solves Triangular solves
are required at initial factorization re nement and up dating As a
result algorithms based on these factorizations may b e exp ensive es
p ecially on parallel computers where triangular solves are exp ensive
In this pap er we prop ose an alternative approach Our new rank
revealing U LV factorization which we call top down U LV factor
ization T D U LV factorization is based on right null vectors of lower
triangular matrices and therefore no triangular solves are required
Right null vectors are easy to estimate accurately using condition esti
mators such as incremental condition estimator ICE The T D U LV
factorization is shown to b e equivalent to the U RV factorization with
the advantage of circumventing triangular solves
Introduction Recent numerical integration metho ds for di erential
algebraic equations D AE s require at each time integration
step the computation of the numerical rank and bases for null spaces of very
large matrices These matrices are obtained by a recursive di erentiation
algorithm which app ends new rows to the previous matrices The pro cess
of incorp orating a new row or column in a matrix is called up dating Other
applications are the solution of underdetermined rank de cient least squares
problems subset selection problems and information re
trieval
The singular value factorization SVD p is known to b e an
extremely reliable to ol for computing the numerical rank and bases for the
null spaces of a matrix However the SVD is to o exp ensive when it comes
to recursive algorithms or real time applications since its computation re
3 1
quires O n ops and the SVD is di cult to up date Therefore
alternative algorithms that are nearly as accurate as the SVD cheaper and
easier to up date are desired
Recently Stewart prop osed two rank revealing factoriza
tions called U LV and U RV factorizations These two factorizations are
1
Here a op is either an addition or a multiplication
e ective in exhibiting the numerical rank and bases for the null spaces The
2
U LV and the U RV factorizations can b e up dated in O n ops sequentially
and in O n ops on an array of n pro cessors Recent work related
to the U RV and U LV factorization b oth in theory and implementation may
b e found in The rank revealing U LV and the U RV al
gorithms are iterative and require estimates of the condition number of some
triangular submatrices at every iteration step of initial factorization re ne
ment and up dating In the U RV and the U LV factorizations small singular
values and asso ciated null vectors are estimated by means of conditions es
timators A survey of condition estimators is given in
In the practical U LV resp U RV factorization however each left resp
right null vector is recomputed from its corresp onding right resp left null
vector via triangular solves Triangular solves are required for the initial
factorization the re nement and up dating For some applications triangular
solves have to b e p erformed many times in order to achieve a required accu
racy Therefore algorithms based on the usual U LV and U RV factorizations
may b e very exp ensive on parallel computers where triangular solves are
exp ensive
For this reason we introduce an alternative rank revealing U LV factor
ization called top down U LV factorization T D U LV factorization This
new factorization relies on right null vectors of lower triangular matrices
which are accurately estimated using condition estimators This results in
circumventing triangular solves required in the usual rank revealing U LV
and U RV factorizations Our T D U LV factorization is essentially equivalent
to the U RV with the advantage of avoiding triangular solves thus it is more
suitable for parallel implementations Furthermore the T D U LV uses the
null vectors obtained from condition estimators in a straithforward way
In this pap er we describ e the T D U LV factorization give an algorithm
to compute it and show how this algorithm can b e implemented re ned and
up dated e ciently The remainder of this pap er is organised as follows In
section we brie y review the usual rank revealing U LV and U RV factor
izations Our new T D U LV factorization metho d is prop osed in section
In section we give details of the T D U LV factorization algorithm The new
algorithm is presented in section Finally we draw a conclusion in section
ULV and URV factorizations In this section we review the rank
revealing U LV and U RV factorizations introduced by Stewart
We rst introduce the concept of numerical rank of a matrix Given a
mn
matrix A R m n a singular value factorization SVD see x
of A has the form
T
A U V
where U u u and V v v are orthogonal matrices and
1 m 1 m
diag is an m n diagonal matrix whose entries the singular
1 n
values of A are ordered such that Then the
1 2 n
numerical rank of A with resp ect to a threshold is de ned as the
number of singular values of A strictly larger than i e
1 k k +1 n
is a threshold b elow which a singular value of the matrix A is declared to
b e numerically null or negligeable The ratio estimates the gap
r +1 k
b etween large and small singular values of A The numerical rank is
well de ned whenever the gap is su ciently large Ways for choosing the
threshold may b e found in
For i k n the columns v of V satisfy kAv k and therefore
i i
are called numerical right nul l vectors k k denotes the matrix norm In
the same way columns u u of U are called numerical left nul l vectors
k +1 n
T
since they satisfy ku Ak for i k n
i
The numerical right nul l space of A is de ned by
r
span fv v g N
k +1 n
k
in the same way we de ne the numerical left nul l space of A by
l
span fu u g N
k +1 n
k
mn
Given a matrix A R a U LV factorization of A has the form
L
k
T
A U V
H E
mm nn k k
with orthogonal matrices U R V R and L R E
k
(m k )(m k ) (m k )k
R lower triangular matrices H R
Such a factorization is said to b e rank revealing if k H E k O and
k +1
L is well conditioned i e L L c where c is some given
k k k 1 k
tolerance
Similarly a U RV factorization of A has the form
R F
k
T
V A U
G
k k (m k )(n k )
where R R G R are upp er triangular matrices and where
k
k (n k )
F R
Such a factorization is said to b e a rank revealing if R is well conditioned
k
T T T
and k F G k O
k +1
In factorizations and the numerical rank of A is revealed by the
dimension of the submatrices L and R resp ectively Orthonormal left and
k k
right bases for the null spaces of A are revealed by the matrix U and V
resp ectively More precisely columns k through n of U and V span the
left and right null spaces of A resp ectively
Factorization and are based on estimating small singular values
of the middle factors L and R and the asso ciated left and right null vectors
resp ectively Then de ating small singular values from the b ottom of ma
trices L and R factorizations and are obtained Adaptive versions
of the U LV and U RV algorithms and results concerning the e ect of esti
mated null vectors on the size of o diagonal blo cks H and F are discussed
in There it is shown that the sizes of H and F dep end strongly on
approximations of the null vectors The norms of H and F in turn a ect
the accuracy of the approximated null spaces A re nement metho d for the
U RV factorization was presented and analysed in Stewart
The usual way to compute a U LV factorization of a matrix A is rst
to compute an ordinary QL factorization of A p and then to p eel
o small singular values one by one from the b ottom of the matrix L
This requires approximations of left null vectors of the triangular ma
trix L at each iteration step of factorization re nement and up dating In
the practical rank revealing U LV factorizations left null vectors are usually
obtained from the corresp onding right null vectors via triangular solves For
very large problems however this results in an extra cost and may lead to
loss of accuracy in the subspaces In the next section we present a more e
cient U LV factorization that avoids triangular solves by working with right
null vectors of lower triangular matrices This reduces the computational
work needed for the triangular solves
TDULV factorization In this section we present the rank revealing
T D U LV factorization The idea of our factorization is to compute rst any
QL factorization of A for example by using the LAPACK routine xGE
2
QLF then to p eel o small singular values of L one after the other from
the top of the matrix L in a sequence of de ation steps until a large singular
value is detected This is achieved by estimating small singular values of L
and asso ciated right null vectors using condition estimators for example by
using the incremental condition estimator ICE implemeted in LAPACK
routine xLAIC This pro cess leads to the so called top down U LV fac
torization T D U LV
E
T
V A U
H L
k
k k (m k )(n k )
where L R E R are lower triangular matrices and where
k
k (n k )
H R
We call such a factorization rank revealing T D U LV factorization if L is
k
T T T
well conditioned and k E H k O
k +1
In the T D U LV factorization the rank of the matrix A is revealed by the
dimension of the right b ottom submatrix L The rst n k columns of the
k
orthogonal matrices U and V furnish orthonormal left and right bases for
null spaces of A resp ectively
We show in the app endix that the rank revealing T D U LV factorization
and the rank revealing U RV factorization are mathematically equiv
alent The advantage of the T D U LV over the U RV is that the T D U LV
works with singular vectors computed by condition estimators in a straith
foward way
Outline of the rank revealing TDULV Algorithm In this section
we discuss the implementation of the T D U LV algorithm We show how to
re ne the factorization to make it rank revealing We then discuss the up dat
ing of the factorization The rank revealing T D U LV factorization pro cess
2
Here the pre x x is S or D
b egins with any QL factorization of A followed by an iteration with makes
the factor L rank revealing The matrix L is declared to b e numerically rank
de cient with resp ect to a threshold if L has at least one singular value
Small singular values of L and asso ciated null vectors v of norm
one are estimated e ciently using condition estimators If the matrix L is
rank de cient then we transform it to an equivalent lower triangular matrix
T
P LQ by means of Givens rotations The orthogonal matrix Q is formed as
the pro duct of Givens rotations such that comp onents of v are annihilated
T
one at a time to obtain the canonical unit vector e i e we have Q v e
1 1
We p ostmultiply L by the orthogonal matrix Q Then we triangularize LQ
T
by premultiplying it by an orhogonal matrix P where P is again formed as
pro duct of Givens rotations It follows that
T T T
kLv k kP LQQ v k kP LQe k
1
T
and hence the rst column of the triangular matrix P LQ is small This way
of pro ceeding is called de ation and applying it rep eatedly the T D U LV is
computed However to obtain accurate null spaces one may have to delay the
de ation and re ne the factorization until the required accuray is achieved
Re nement Factorization reveals the numerical rank of A by the
dimension of the matrix L Bases for approximate left and right null spaces
k
of A are given by the rst n k columns of U and V resp ectively To obtain an
accurate basis for the left null space one may need to re ne the factorization
by bringing the matrix E to near diagonal form This is achieved by Givens
rotations Supp ose we have obtained the partial factorization
e
T
V A U
h L
n 1
where the matrix in the middle is assumed to b e rank de cient
The aim of the re nement is to make the norm khk where is some
de ation parameter This leads to accurate bases for null spaces of A and
T
A The rst step in the re nement is to compute an orthogonal matrix Q
such that o diagonal elements of the rst column of LQ vanish i e
0 0T
e h
LQ
0
L
n 1
This is achieved by zeroing successively elements of h by means of Givens
rotations The matrix Q is the pro duct of these Givens rotations applied to
L from the right Nonzero elements then app ear in the rst row of LQ The
second step in the re nement is to determine an orthogonal matrix P such
0T T
that element of h are annihilated by premultiplying LQ by P This is done
0T
by zeroing successively the elements of h by means of Givens rotations We
then obtain the following lower triangular matrix
00
e
T
P LQ
00 00
h L
n 1
00
After these two steps of re nement elements of h have b ecome smaller If
00
kh k then we de ate the rst row and column in To maintain
null spaces transformations P and Q in these two steps of re nement are
also applied to U and V At this p oint the factorization of A is given by
00
e
T
VQ A UP
00 00
h L
n 1
In this fashion the matrix E in is made closer to a diagonal matrix
TDULV Updating The T D U LV factorization can b e up dated when
a new row is incorp orated at the b ottom of the matrix A Assume that after
having computed a rank revealing factorization of A we wish to include
a new row in A The aim of the up dating is to compute a rank revealing
factorization of the up dated matrix from that of A at a low computational
2
cost namely O n or less This should b e done without destroying small
elements of E and F The row updating of the rank revealing T D U LV is
describ ed as follows
E
U A
C B
T
V H L
A
k
T
a
T T
x y
T T T T
where a is the app ended row and where x y a V
In the rst phase of up dating we annihilate the rst n k comp onents
T
of x while maintaining the triangular form of E This is p erformed by
applying a sequence of interleaved right and left Givens rotations In the
pro cess each right rotation introduces ab ove the diagonal of E a nonzero
element which is zero ed out by left rotation In this annihilation pro cess
T
of x only rows of E and the rst n k columns of the middle matrix
in are involved In this fashion smallness of matrices E and H is
T
preserved The reduction pro cedure where only E and x are shown is
illustrated in F ig In all gures vertical arrows p oint out the columns
involved in a p ostmultiplication by a rotation Horizontal arrows p oint out
the rows involved in a premultiplication by a rotation A check over an
element indicates the element to b e eliminated
e e e e
e e e e e e
e e e e e e e e e
e e e e e e e e e e e e
x x x x x x x x x x
e e e e
e e e e e e e e e
e e e e e e e e e e e e
e e e e e e e e e e e e e e e e
x x x x x x
T
Fig Annihilation of components of x
The second phase is to triangularize the following trap ezoidal matrix by
annihilating the last row
e
h l
h l l
h l l l
h l l l l
x y y y y
This is p erformed by means of Givens rotations as follows
e
e e
h l
h l h l
h l l
h l l h l l
h l l l h l l l
h l l l l
h l l l l h l l l l
h l l l l
x y y y x y y y y
x y y y
e
e
e
h l
h l
h l l
h l l l
h l l
h l l
h l l l
h l l l
h l l l
h l l l l
h l l l l
h l l l l
x y y
x y y
x y
x e e
x l x l h l
h l l h l l h l l
h l l l h l l l h l l l
h l l l l h l l l l h l l l l
x x y
Fig Triangularization
The triangularization pro cess replaces the zeros in the last matrix of F ig by
some small elements h These small elements can b e eliminated or neglected
TDULV Algorithm
The rank revealing T D U LV factorization is summarized in the following
algorithm
Input
mn
Matrix A R m n to b e decomp osed
Threshold for singular values of A
De ation tolerance for kH k
Maximum number of iterations N for the re nement
Output
Numerical rank k
mm nn
Orthogonal matrices U R V R and a lower triangular
nn
matrix L R
0
where Q is orthogonal Compute a QL factorization of A A Q
L
nn
and L R is lower triangular e g using the LAPACK routine
xGEQLF
Initialization U Q V I k n and itstep
n
Compute the smallest singular value of L and the asso ciated right
n
n n
null vector v R of norm one e g by using the incremental condi
tion estimator ICE implemented in LAPACK routines xLAIC
While and k do
k
While itstep N do
For j n n k do
k k
Determine a Givens rotation Q R so that
j j +1
k T
zero es comp onent premultiplication of v by Q
j j +1
k k k
v of v using v Up date L and V
j +1 j
I I
n k n k
and V V L L
Q Q
j j +1 j j +1
k k
Determine a Givens rotation P R so that pre
j j +1
multiplication of L L n k n n k n
k
T
zero es l using l Up date L and U by P
j j +1 j +1 j +1
j j +1
I
I
n k
m k
L L and U U
T
P
P
j j +1
j j +1
Enddo
If kL n k n n k k or itstep N then
De ation Set k k and itstep
Else
Re nement Set itstep itstep
Determine a sequence of Givens rotations
Q Q so that p ostmultiplication
n+1 k n+2 k n+1 k n
of L by Q Q Q zero es the
k k n+1 k n+2 k n+1 k n
elements L n k n n k Up date L and V
I I
n k n k
and V V L L
Q Q
k k
Determine a sequence of Givens rotations
P P so that premultiplication of L
n+1 k n n+1 k n+2 k k
T
by P where P P P zero es the
k n+1 k n n+1 k n+2 k
k
elements L n k n k n Up date L and U
I
I
n k
m k
L L and U U
T
P P
k
k
Endif
Compute the smallest singular value of L and the
k k
k k
asso ciated right null vector v R of norm one
Endwhile
Endwhile
End of TDULV Algorithm
The algorithm terminates if a lower b ound is computed The dimen
k
sion k of the b ottom right matrix L is equal to the numerical rank Bases
k
for left and right null spaces are given by the rst n k columns of U
and V resp ectively
Example
We now describ e the steps for the n k th stage of the algorithm and
illustrate it for the case k At this stage already n de ations have
b een p erformed and the matrix L has the form where the right b ottom
submatrix L has dimension k According to the ab ove algorithm only
4
the matrix L is involved in the coming steps we therefore sketch only this
4
4
matrix The next step in our algorithm is to compute and v R approx
imations of the smallest singular value of L and the asso ciated right null
4
vector Then we annihilate successively the th r d and the nd comp onent
T
T T
of v using Givens rotations Q Q Q so that
12 23 34
T T T T
v Q Q Q
34 23 12
The sketch b elow clari es the e ect of carring out successive transformations
Q on v
i i+1
v v v
C B C C B C B B
T T T
Q Q Q
v v v
C B C C B C B B
23 12 34
C B C C B C B B
A A A A
v v
v
Fig Reduction of v
We must then p ostmultiply L by these rotations This multiplication by
4
Q pro duces a nonzero i i entry in L To restore the triangular
i i+1 4
T
form to L we premultiply it by some appropriate plane rotation P For
4
i i+1
i we then have
T T T T T
L L Q P L Q P L Q Q P P P L Q Q Q
4 4 34 4 34 4 34 23 4 34 23 12
34 34 12 23 34
The triangular form changes as follows
l l
l l
l l
l l l l
l l l
l l l l l l
l l l l l l l
l l l l l l l l
l l l l l l l l
e l
l l
h l l l
l l
h l l l l l
l l l
h l l l l l l l
l l l l
Fig De ation procedure
The elements h and e in the rst column of L indicate that small elements
4
have b een generated in this column To see this consider the norm of L v
4
T T T
kL v k kP L QQ v k kP L Qe k
4 4 4 4 1
where P P P P and Q Q Q Q Thus the rst column of the
34 23 12 34 23 12
T
triangular matrix P L Q is small One can stop at this p oint and take the
4
computed factorization as rank revealing factorization However to obtain
the correct numerical rank and accurate null spaces one has to bring the
matrix E to near diagonal form by reducing the norm of H This is acom
T
plished by reducing the size of elements h in the rst column of P L Q in
4
each de ation step as follows
e e h
l
h l
h l l h l l
h l l l h l l l
e h h h
e h h
l
l
l l
l l
l l l
h l l l
Fig Zeroing o diagonal elements of the rst column
We reduce now the rst row of the matrix L using left rotations as follows
e h h e h h h
l l
l l l l
h l l l l l l
e
e h
h l
l
h l l
h l l
h l l l
h l l l
Fig Zeroing o diagonal elements of the rst row
Example
Let A b e the lower triangular matrix with on the diagonal and as o
diagonal elements For m n and the rank is and the middle
matrix L is
C B
e
A
e e
Tables and show that the null spaces computed from the T D U LV closely
approximate the null spaces computed from the SVD
U U
T D U LV SVD
Table Bases for the left nul l space
V V
T D U LV SVD
Table Bases for the right nul l space
Conclusion In this pap er we have prop osed a new U LV factorization
called T D U LV factorization and an algorithm to compute it This factoriza
tion is based on right null vectors of lower triangular matrices rather than left
null vectors as in the U LV factorizations First this has resulted in avoiding
triangular solves which may b e exp ensive on parallel computers and reduc
ing the cost related to these solves esp ecially if they have to b e p erformed
many times with very large matrices Second this avoids including parame
ters related to triangular solves Furthermore our metho d uses null vectors
computed by condition estimators in a straightforward way Therefore it may
b e more accurate than the U RV in exhibing the numerical rank and bases
for null spaces
App endix
LEMMA
Rank revealing U RV factorizations and T D U LV factorizations of a matrix
A are equivalent
PROOF
Let A have the rank revealing T D U LV factorization
E
T
V A U
H L
k
Then we can write
E
T
J VJ A UJ J
n n m m
H L
k
where
J J
n k k
J J
n m
J J
k m k
and where J denotes the k k ip matrix Therefore
k
J L J J HJ
T
k k k k n k
VJ A UJ
n m
J EJ
m k n k
which is of the form
R F
T
A U V
G
with
G J EJ F J HJ R J L J U UJ V VJ
m k n k k n k k k k m n
Note that the matrices G and R are upp er triangular and U and V are or
thogonal Furthermore we have
1 1
k k kL kGk kE k kF k kH k kR
k k
therefore the T D U RV factorization is rank revealing if and only if the U RV
is rank revealing The bases obtained from the T D U LV are given by the
rst columns of the orthogonal matrices UJ and VJ
m n
References
M W Berry S Dumais And G O Brien The computational com
plexity of alternative updating approaches for an SVD encoded indexing
scheme In Pro ceedings of the Seventh SIAM Conference of Parallel Pro
cessing for Sci Comput SIAM Philadelphia
M W Berry And R D Fierro Low rank orthogonal decomposition
for retrieval applications Num Linear Alg Appl
C H Bischof Incremental condition estimator SIAM J Matrix Anal
Appl
C H Bischof J G Lewis And D J Pierce Incremental con
dition estimation for sparse matrices SIAM J Matrix Anal Appl
C H Bischof And P T P Tang Generalizing incremental con
dition estimation J Numer Linear Algebra Appl
J R Bunch And N P Nielsen Updating the SVD Numer Math
T F Coleman Large sparse numerical optimization Lecture Notes
in Computer Science Sringer Verlag New York
R D Fierro Perturbation analysis for tow sided or complete orthog
onal decompositions SIAM J Matrix Anal Appl
R D Fierro And P C Hansen Low rank revealing two sided
orthogonal decompositions Tech Rep ort Department of Mathe
matics California State Univerisity San Marcos CA Octob er
R D Fierro And J R Bunch Bounding the subspaces from rank
revealing two sided orthogonal decompositions SIAM J Matrix Anal
Appl
R D Fierro And P C Hansen Accuracy of TSVD solutions com
puted from rank revealing decompositions Numer Math
G H Golub Numerical methods for solving linear least squares prob
lems Numer Math
G H Golub V Klema And G W Stewart Rank degeneracy
and least squares problems Tech Rep ort TR Department of Com
puter Science University of Mayland
G H Golub And C V Loan Matrix computations John Hopkins
Baltimore Second Edition
W W Hager Condition estimates SIAM J Sci Statist Comput
N J Higham A survey of condition number estimation for triangular
matrices SIAM Review
P Kunkel V Mehrmann A new class of discretization methods
for the solution of linear di erential algebraic equations with variable
coe cients SIAM J Numer Anal
P Kunkel V Mehrmann Local and global invariants of linear
di erential algebraic equations and their relations Electr Trans Nu
mer Analysis to app ear
P Kunkel V Mehrmann W Rath And J Weickert A new
software package for linear di erential algebraic equations SIAM J Sci
Comput
C L Lawson And R J Hanson Solving least squares problems
Prentice Hall Englewoo d Cli s NJ
K J R Liu D P O Leary G W Stewart And Y J J Wu
URV ESPRIT for tracking time varying signals IEEE Trans Signal
Pro c
R Mathias And G W Stewart A block QR algorithm and the
singular value decomposition Linear Algebra Appl
H Park And L Elden Downdating the rank revealing URV decom
positions SIAM J Matrix Anal Appl
D J Pierce And R J Plemmons Fast adaptive condition estima
tion SIAM J Matrix Anal Appl
G W Stewart On algorithm for re ning a rank revealing URV de
composition and a perturbation theorem for singular values Tech Rep ort
CS TR University of Mayland Department of Computer Science
College Park MD
G W Stewart An updating algorithm for for subspace tracking
IEEE Trans Signal Pro c
G W Stewart Updating a rank revealing ULV decomposition SIAM
J Matrix Anal Appl
G W Stewart Determining rank in presence of errors pp
in M S Mo onen G H Golub And B L R De Mo or Eds Linear
Algebra for Large Scale and Real time Applications NATO ASI Series
Kluwer Dordrecht
G W Stewart Updating URV decompositions in parallel Parallel
Comp
P T P Tang Dynamic condition estimation and Rayleigh Ritz ap
proximation SIAM J Matrix Anal Appl
B N PARLETT The symmetric eigenvalue problem Prentice Hall
Englewoo d Cli s N J