A HISTORY OF INVERSE ITERATION
y
ILSE C F IPSEN
The Metho d Inverse iteration is a metho d for computing eigenfunctions of
linear op erators Here we consider inverse iteration for the solution of the algebraic
eigenvalue problem
Ax x
where A is a real or complex square matrix the scalar is an eigenvalue and the non
zero vector x is an eigenvector Inverse iteration is the metho d of choice when one has
at one s disp osal approximations to a sp eci ed subset of eigenvalues of A for which one
wants to compute asso ciated eigenvectors The metho d is frequently used in structural
mechanics for instance when the extreme eigenvalues and corresp onding eigenvectors
of Hermitian p ositive semi de nite eigenproblems are sought
Given an approximation to an eigenvalue of A inverse iteration generates a
1
sequence of vectors x from a given starting vector x by solving the linear systems
k 0
A I x s x k
k k k 1
The scalar s is resp onsible for normalising x Usually one choses s so that kx k
k k k k
in some norm If everything go es well the sequence of iterates x converges to
k
an eigenvector asso ciated with an eigenvalue closest to In particular when there
is only a single eigenvalue closest to then the x converge to an eigenvector
k
asso ciated with provided the starting vector x contains a contribution of this
0
k
eigenvector Since x is a multiple of the vector A I y and since
k 0
1
is the largest eigenvalue of A I the contribution in x of the eigendirection
k
asso ciated with increases faster as k than the contributions asso ciated with the
2
other eigenvalues Inverse iteration is mathematically identical to the p ower metho d
1
applied to A I
In Wilkinson remarked Inverse iteration is now the most widely used
metho d for computing eigenvectors corresp onding to selected eigenvalues which have
already b een computed more or less accurately p A lo ok at software in the
public domain shows that this is still true to day
Yet after so many years the b ehaviour of the metho d is still not completely
understo o d neither in exact nor in oating p oint arithmetic As a consequence
even highly sophisticated state of the art software like LAPACK still comes across
matrices for which it cannot compute eigenvectors to high accuracy without a drastic
sacri ce in e ciency
3
This article describ es the e orts undertaken on b ehalf of inverse iteration from its
To app ear in Helmut Wielandt Mathematische Werke Mathematical Works Volume II Ma
trix Theory and Analysis Hupp ert B and Schneider H eds Walter de Gruyter Berlin
y
Department of Mathematics North Carolina State University P O Box Raleigh NC
USA ipsen math ncsu edu This work was supp orted in part by NSF grant CCR
1
Here I denotes the identity matrix
2
Mathematical identity means that equality holds in exact arithmetic but not necessarily in
nite precision arithmetic b ecause the two metho ds determine their iterates by means of di erent
op erations
3
We concentrate on the solution of the ordinary eigenvalue problem rather than the generalised
eigenvalue problem Ax B x and we do not discuss techniques for convergence acceleration such
as Rayleigh quotient iteration where the shift changes with each iteration
inception to the present day Octob er Here are a few of the issues that prompted
these e orts The iterates may not converge to an eigenvector b ecause the starting
vector x do es not have a contribution in the desired eigenvector or b ecause there
0
may b e several di erent eigenvalues of A at the same distance from The eigenvalue
closest to may b e a multiple eigenvalue and inverse iteration may have to compute a
basis for the corresp onding invariant subspace Unless linear indep endence is enforced
explicitly inverse iteration usually fails to compute the desired basis When the matrix
is Hermitian linear indep endence is enforced by orthogonalising an iterate against the
already computed eigenvectors But dep ending on the shift and the starting vector
x iterates may still fail to converge The problem b ecomes even more pronounced
0
when happ ens to approximate a cluster of close but not identical eigenvalues Should
one treat the cluster like a collection of separate eigenvalues or like a single multiple
eigenvalue or like several multiple eigenvalues How close together must eigenvalues
b e to qualify as a cluster Should an iterate b e orthogonalised against all eigenvectors
in the cluster or against just a few
Wielandt Started It Inverse iteration was intro duced by Wielandt in
Although Peters and Wilkinson remark p without further elab oration
that a numb er of p eople seem to have had the idea indep endently Wielandt is usually
the one credited with the intro duction of inverse iteration He
refers to inverse iteration as fractional iteration gebro chene Iteration in German
1
b ecause the matrix A I is a fractional linear function of A p He
p oints out the b ene t of inverse iteration in the stability analysis of vibrating systems
that are small p erturbations of systems whose b ehaviour is known xI In this
case go o d approximations to the eigenvalues of the p erturb ed system are available
The computational asp ects of Wielandt s work leading to the development of inverse
iteration are describ ed in the companion pap er
Adapted to the algebraic eigenvalue problem Wielandt s justi cation for inverse
iteration lo oks as follows xI I I c Cho ose so that Then Ax x
b ecomes
A I x x
Replacing x on the left by x and on the right by x and writing s instead of
k k 1 k
gives inverse iteration
In Crandall presents inverse iteration for real symmetric matrices as an
4
instance of a particular relaxation metho d He do es not reference Wielandt s pap er
Wilkinson s Idea The p erson most closely asso ciated with inverse iteration
for the algebraic eigenvalue problem is Jim Wilkinson Although Wilkinson gives
the credit for inverse iteration to Wielandt he himself came up with
it indep endently in as a result of trying to improve Givens s metho d for
computing a single eigenvector of a symmetric tridiagonal matrix
We mo dify Wilkinson s idea slightly and present it for a general matrix A rather
than just a symmetric tridiagonal matrix Supp ose A has order n and the shift is
4
Crandall page gives the following characterisation of relaxation metho ds A unique
feature of relaxation metho ds is that they are approximate pro cedures which are not rigidly prescrib ed
in advance The precise pro cedure to b e followed is not dictated but is left to the intuition and
accumulated exp erience of the computer The computer s intelligence is thus an active or dynamic
link in the computational chain It is this fact which has made relaxation so attractive to many
computers Thus at the time Crandall wrote his pap er a computer was still a human b eing and
for exploratory purp oses the relaxation metho ds were carried out with a slide rule p
closer to the simple eigenvalue than to any other eigenvalue of A Wilkinson s idea
is the following Since an eigenvector x is determined up to a multiple by n of the
n equations A I x approximate x by a non zero vector z that solves n
equations of A I z
Why should such a z b e a go o d approximation to x Partition A so as to distin
guish its leading principal submatrix A of order n and partition z conformally
A a z
1
A z
a
2
Supp ose is also not an eigenvalue of A so A I is non singular If we x to a
non zero value then there exists a solution z to the smaller system
A I z a
1
5
This implies that z is a non zero solution to the larger system
A I z e where a z
n 2
If e contains a contribution of an eigenvector x then this contribution is ampli ed
n
in z by Since is the single closest eigenvalue to the contributions of all
other eigenvectors in z are ampli ed by smaller amounts Thus z is closer to x than
is e
n
Instead of solving the rst n equations one could solve any set of n equations
If the ith equation is the one that is omitted then the right hand side is a multiple
of e In general the right hand side can b e any vector x as long as it contains a
i 0
contribution of a desired eigenvector x Then the solution x of A I x x is
1 1 0
closer to x than is x Hence as inverse iteration progresses the iterates x converge
0 k
to an eigenvector x provided the asso ciated eigenvalue is simple
Wilkinson Takes Charge The foremost issues on Wilkinson s mind were
the choice of appropriate starting vectors and the misfortune of having to solve a
linear system that is highly ill conditioned when is an accurate approximation to an
eigenvalue of A
From to Wilkinson analysed the computation of a single eigenvector
of a symmetric tridiagonal matrix determination of the convergence rate in exact
arithmetic choice of a go o d starting vector in exact arithmetic and e ect of the
backward error due to the solution of the linear system by Gaussian elimination with
partial pivoting on the size of the iterate and the convergence rate pp
pp Regarding the computation of eigenvectors asso ciated with almost
coincident eigenvalues he recommends a di erent p erturbation of the shift for each
eigenvalue and the use of the Gram Schmidt algorithm to orthogonalise the current
iterate against previously computed eigenvectors p
Wilkinson applies a similar treatment to computing a single eigenvector when the
eigenvalue is semisimple and the matrix is real and nonsymmetric pp As
Wielandt had already observed in the context of the p ower metho d Wilkinson realises
that the iterates do not necessarily improve in successive iterations when the desired
eigenvalue is almost defective In particular Wilkinson analyses the e ect of the
backward error due to the solution of the linear system on the size of the iterate and
the residual He also gives a roundo error analysis for the computation of complex
eigenvectors of real Hessenb erg matrices in real arithmetic In the same spirit Varah
5
The vector e denotes the ith column of the identity matrix i
analyses the computation of a single eigenvector for a complex Hessenb erg matrix
and presents p erturbation and roundo error analyses for well separated eigenvalues
in
Software on the Horizon Ab out six years later in Peters and Wilkin
son publish ALGOL subroutines that p erform inverse iteration on three classes
of matrices real symmetric tridiagonal real upp er Hessenb erg and complex upp er
Hessenb erg They recommend that these subroutines b e used only if no more than
twenty ve p ercent of the eigenvectors are desired
Shortly thereafter app ears EISPACK a collection of FORTRAN sub
routines for the computation of eigenvalues and eigenvectors that is based up on the
ALGOL subroutines in the EISPACK story is told in along with some of
the di culties in translating the subroutines from ALGOL to FORTRAN EISPACK
contains essentially the same subroutines for inverse iteration as its ALGOL predeces
sor
On the mathematical side Wilkinson shows that it is not to o di cult to nd
6
go o d starting vectors If the shift is a go o d approximation to an eigenvalue of A
then inspite of a backward error from linear system solution one can always nd a x
0
7
for which x is a go o d approximation to an eigenvector of A x In particular
1
among the vectors of any orthogonal basis there is at least one that constitutes a go o d
starting vector p
Wilkinson also considers the computation in exact arithmetic of a single eigenvec
8
tor when the eigenvalue is ill conditioned and when the eigenvalue is defective
He remarks with regard to a single iteration p that even in the case of
an ill conditioned eigenvalue the extraordinary ability of inverse iteration to pro duce
an exact eigenvector of a neighb ouring matrix corresp onding to a given eigenvalue is
immensely valuable and reassuring in practice
In Parlett and Po ole present a geometric theory of inverse itera
tion without orthogonalisation This theory applies to general complex matrices and
provides conditions for the convergence of starting subspaces to invariant subspaces
Watkins resumes this geometric p oint of view nearly twenty years later
In his last pap er on inverse iteration considered by many to b e the de nitive
treatment Wilkinson argues that there is no danger in solving an ill conditioned sys
tem He demonstrates that even though the computed solution may b e totally wrong
it is still likely to have the right direction and that is all we need He also shows that
to rst order inverse iteration is related to Newton s metho d for solving non linear
systems
At last there is a blo ck version of inverse iteration called subspace iteration
or simultaneous iteration that iterates on several vectors simultaneously and enforces
linear indep endence of the vectors from time to time Chatelin shows that
subspace iteration is unstable when used to compute a basis of an invariant subspace
asso ciated with a defective eigenvalue b ecause the basis consists of principal vectors
in addition to eigenvectors x She presents a numerically stable mo di ed
Newton s metho d to compute the basis of such defective invariant subspaces
6
Wilkinson means go o d in the backward sense is an eigenvalue of a matrix close to A p
7
As b efore this means go o d in the backward sense x is an eigenvector corresp onding to the
1
eigenvalue of a matrix close to A
8
A simple eigenvalue is called ill conditioned if the acute angle b etween its left and right eigen
vectors is large x I I I xx
The New Age After several years of relative calm there is renewed interest
in inverse iteration Jessup and Ipsen take up the case of starting vectors They
p erform a statistical analysis to justify the choice of random vectors as starting vec
tors In order to improve the accuracy of TINVIT the EISPACK implementation of
inverse iteration for real symmetric tridiagonal matrices they mo dify three features
of TINVIT choice of starting vectors convergence criterion and the decision of when
and which iterates to orthogonalise The desire for improving the accuracy came as a
result of comparing parallel implementations of Cupp en s divide and conquer metho d
and inverse iteration While the parallel implementation of TINVIT with bi
section in turned out to b e the fastest metho d on an INTEL iPSC hyp ercub e it
was not as accurate as Cupp en s metho d The changes in increase the numerical
accuracy of TINVIT to that of Cupp en s metho d for matrices of order up to
A more recent parallel implementation of inverse iteration for real symmetric
tridiagonal matrices on the MasPar SP solves the linear system rst by a fast
parallel metho d such as cyclic reduction If the solution is not accurate enough the
system is subsequently solved by a more accurate p ossibly slower metho d such as
Gaussian elimination with partial pivoting
The collection of FORTRAN subroutines in LAPACK for solving eigenvalue
problems is designed to replace the EISPACK routines LAPACK contains subroutines
for inverse iteration on symmetric tridiagonal matrices and real and complex upp er
Hessenb erg matrices
Demmel and Veselic derive relative error b ounds as opp osed to the traditional
absolute error b ounds for eigenvalues and eigenvectors of symmetric p ositive de nite
matrices for the case of comp onentwise relative p erturbations They conclude that
inverse iteration with bisection when applied to the original matrix gives b etter error
b ounds than a metho d that requires a preliminary transformation of the matrix to
tridiagonal form
Combinations of Rayleigh quotient and inverse iteration are explored in
Not All Is Well In spite of all the e orts devoted to the development of
inverse iteration the metho d still has its pitfalls In corrob oration we give two quotes
from The Algebraic Eigenvalue Problem which thirty years after its rst publica
tion is still the bible for eigenvalue problems
With regard to the loss of orthogonality in the computation of eigenvectors asso
ciated with nearly coincident eigenvalues of symmetric tridiagonal matrices and the
necessity to orthogonalise iterates against previously computed eigenvectors Wilkin
son says p Of course we cannot exp ect the computed vectors to b e orthog
onal but rememb er that deterioration of orthogonality sets in well b efore we reach
pathologically close eigenvalues It must b e admitted though that this solution to the
problem is somewhat inelegant It has the disadvantage that if we require orthogonal
vectors we must apply the Schmidt orthogonalization pro cess to the computed vectors
Also unless the pro cess can b e put on a sounder fo oting there remains the danger
that we shall not have full digital information ab out the subspace
Later on he concludes p Inverse iteration gives a very satisfactory
solution to the problem as far as reasonably well separated eigenvalues are concerned
The problem of determining reliably full digital information in the subspace spanned
by eigenvectors corresp onding to coincident or pathologically close eigenvalues has
never b een satisfactorily solved
Unfortunately this is still true to day Compared to other metho ds current imple
mentations of inverse iteration sometimes give a lower accuracy or require more work
to attain the same accuracy when eigenvalues are not well separated or when they o c
cur in large clusters Because the criteria are heuristic for deciding when eigenvalues
are su ciently close to warrant orthogonalisation an iterate may b e orthogonalised
against to o many or to o few previously computed eigenvectors
Part of the problem is that inverse iteration is not as easy to analyse as the p ower
metho d due to the nonlinear o ccurrence of the shift x Most existing analyses
concentrate on the nite precision b ehaviour of inverse iteration But lots of things
can wrong in exact arithmetic Even when the matrix is Hermitian and the numb er
of iterations is allowed to b e in nite an unwisely chosen shift can result in a failure to
compute all desired eigenvectors or in the pairing up of eigenvectors with the wrong
eigenvalues The choice of an appropriate shift b ecomes even more delicate when
the numb er of iterations is nite even though the arithmetic is still exact Therefore
it is most imp ortant to carefully separate the prop erties of inverse iteration in exact
arithmetic from those in oating p oint arithmetic Chandrasekaran and Ipsen
pursue this approach rigorously and derive criteria for shift selection orthogonalisation
of iterates and against previously computed eigenvectors and convergence criteria The
goal is a numerically robust and provably accurate inverse iteration algorithm
Acknowledgement I thank Hans Schneider for inviting me to write this contri
bution and for his generous and lucid advice Stan Eisenstat Tim Kelley Carl Meyer
and esp ecially Fernando Reitich provided many suggestions that greatly improved the
quality of the pap er
REFERENCES
E Anderson Z Bai C Bischof J Demmel J Dongarra J Du Croz A Green
baum S Hammarling A McKenney S Oustrouchov and D Sorensen LAPACK
Users Guide So ciety for Industrial and Applied Mathematics Philadelphia
K Bathe and E Wilson Discussion of a paper by K K Gupta Int J Num Meth Engng
pp
Numerical Methods in Finite Element Analysis Prentice Hall Englewo o d Cli s NJ
S Chandrasekaran and I Ipsen Inverse iteration for symmetric matrices Manuscript in
Preparation
F Chatelin Il l conditioned eigenproblems in Large Scale Eigenvalue Problems North
Holland Amsterdam pp
Valeurs Propres de Matrices Masson Paris
S Crandall Iterative procedures related to relaxation methods for eigenvalue problems Pro
ceedings of the Royal So ciety of London Series A pp
On a relaxation method for eigenvalue problems Journal of Mathematical Physics
pp
J Cuppen A divide and conquer method for the symmetric tridiagonal eigenproblem Num
Math pp
J Demmel and K Veselic Jacobi s method is more accurate than QR SIAM J Matrix
Anal Appl pp
P Deuflhard and A Hohmann Numerische Mathematik I Au age Walter de Gruyter
Berlin
K Gupta Eigenproblem solution by a combined Sturm sequence and inverse iteration tech
nique Int J Num Meth Engng pp
Development of a uni ed numerical procedure for free vibration analysis of structures
Int J Num Meth Engng pp
R Hanson and G Luecke Paral lel inverse iteration for eigenvalue and singular value de
compositions SIAM News pp
A Householder The Theory of Matrices in Numerical Analysis Dover New York
I Ipsen Helmut Wielandt s contributions to the numerical solution of complex eigenvalue
problems in Helmut Wielandt Mathematische Werke Mathematical Works B Hupp ert
and H Schneider eds vol I I Matrix Theory and Analysis Walter de Gruyter Berlin
I Ipsen and E Jessup Solving the symmetric tridiagonal eigenvalue problem on the hyper
cube SIAM J Sci Stat Comput pp
P Jensen The solution of large symmetric eigenvalue problems by sectioning SIAM J Numer
Anal pp
An inclusion theorem related to inverse iteration Linear Algebra and its Applications
pp
E Jessup and I Ipsen Improving the accuracy of inverse iteration SIAM J Sci Stat
Comput pp
S Ma M Patrick and D Szyld A paral lel hybrid algorithm for the generalized eigen
problem in Parallel Pro cessing for Scienti c Computing So ciety for Industrial and Applied
Mathematics Philadelphia pp
B Parlett The Symmetric Eigenvalue Problem Prentice Hall Englewo o d Cli s NJ
The software scene in the extraction of eigenvalues from sparse matrices SIAM J Sci
Stat Comput pp
B Parlett and W Poole A geometric theory for the QR LU and power iterations SIAM
J Numer Anal pp
G Peters and J Wilkinson The calculation of speci ed eigenvectors by inverse iteration
contribution II in Linear Algebra Handb o ok for Automatic Computation Volume I I
Springer Verlag Berlin pp
Inverse iteration il l conditioned equations and Newton s method SIAM Review
pp
Y Saad Numerical Methods for Large Eigenvalue Problems Manchester University Press
New York
B Smith J Boyle J Dongarra B Garbow Y Ikebe V Klema and C Moler
Matrix Eigensystem Routines EISPACK Guide nd Ed Springer Verlag Berlin
B Smith J Boyle Y Ikebe V Klema and C Moler Matrix Eigensystem Routines
EISPACK Guide Springer Verlag Berlin
M Smith and S Huttin Frequency modi cation using Newton s method and inverse iteration
eigenvector updating AIAA Journal pp
G Stewart Introduction to Matrix Computations Academic Press New York
E Stiefel Einfuhrung in die Numerische Mathematik Teubner Stuttgart
D Szyld Criteria for combining inverse and Rayleigh quotient iteration SIAM J Numer
Anal pp
J Varah The calculation of the eigenvectors of a general complex matrix by inverse iteration
Math Comp pp
D Watkins Understanding the QR algorithm SIAM Review pp
H Wielandt Beitr age zur mathematischen Behand lung komplexer Eigenwertprobleme
Teil III Das Iterationsverfahren in der Flatterrechnung Bericht B J Aero dynami
sche Versuchsanstalt G ottingen Germany pages
Beitr age zur mathematischen Behand lung komplexer Eigenwertprobleme Teil V Bes
timmung h oherer Eigenwerte durch gebrochene Iteration Bericht B J Aero dynami
sche Versuchsanstalt G ottingen Germany pages
J Wilkinson The calculation of the eigenvectors of codiagonal matrices Computer J
pp
Rounding Errors in Algebraic Processes Prentice Hall Englewo o d Cli s NJ
The Algebraic Eigenvalue Problem Clarendon Press Oxford
Inverse iteration in theory and practice in Symp osia Matematica vol X Institutio
Nationale di Alta Matematica Monograf Bologna pp
Note on inverse iteration and il l conditioned eigensystems in Acta Universitatis Caroli
nae Mathematica et Physica No Universita Karlova Fakulta Matematiky a Fyziky
pp
J Wilkinson and C Reinsch Contribution II The calculation of speci ed eigenvectors
by inverse iteration in Handb o ok for Automatic Computation Volume II Linear Algebra
Springer Verlag Berlin pp