Accurate and Efficient Matrix Computations with Totally Positive Matrices

Plamen Koev Department of Mathematics M.I.T.

SIAM Annual Meeting, Montreal, June, 2003 Goals

• Accurate and Efficient Linear Algebra with Totally Positive Matrices – Linear Equation Solving, Inversion – Eigenvalues, Eigenvectors – Singular Values, Singular Vectors • Efficient means O(n3) – Usual arithmetic model fl(a b) = (a b)(1 + δ), |δ| ≤  • Accurate means correct sign and leading digits

|λˆi − λi| ≤ O()|λi|, |σˆi − σi| ≤ O()σi, – Contrast: Traditional algorithms ||A|| |λˆi − λi| ≤ O() , |σˆi − σi| ≤ O()σ1 |y∗x| Example: 100 × 100 Hilbert Matrix H = 1/(i + j)

Singular values of Hilb(100), blue = accurate, red = usual 0 10

−20 10

−40 10

−60 10

−80 10

−100 10

−120 10

−140 10

0 10 20 30 40 50 60 70 80 90 100 How do we lose accuracy in floating point?

• fl(a ⊗ b) = (a ⊗ b)(1 + δ), no under/overflow • How can we lose accuracy in computing in floating point? – OK to multiply, divide, add positive numbers Proof: 1 + δ factors can be factored out

– xi ± xj, where xi and xj are initial data (so exact) – Cancellation when subtracting approximate results dangerous: .12345xxx - .12345yyy .00000zzz • Trick: We will only compute:

– 1/(1 + xy); 1/x; xy; 1/(xi − yj); x, y > 0 – never subtract approximate quantities Bidiagonal Matrices

• Demmel–Kahan ’91    a1 b1         a2 b2       .. ..  B =  . .         an−1 bn−1      an

• ai, bi determine σi accurately • dqds preserves the relative accuracy T 2 • Eigenvalues of SPD tridiagonal matrices: λi(B B) = σi (B) – computable accurately given the Cholesky factors Total Positivity (TP)

• Def: A is TP if all minors > 0 – Hilbert H = 1/(i + j)

– Cauchy C = 1/(xi + yj), if 0 < xi, yj-increasing aj – Generalized Vandermonde V = xi , 0 < xi, ai - increasing – TP·TP – Schur-complement(TP) – diag(±1)·TP·diag(±1) – See books by Karlin, Gantmacher, Krein • Theory extends to Totally Nonnegative, Oscillatory, Sign-regular • Motivation: – Eigenvalues and SVD well determined – Right answer costs O(n3), same as the wrong • Trick: choose the right parametrization: Bidiagonal Total Positivity and Neville Elimination – 1

• Gaussian Elimination    1 1 1 1         1 2 4 8         1 3 9 27     1 4 16 64  Total Positivity and Neville Elimination – 2

• Gaussian Elimination    1 1 1 1         1 2 4 8         1 3 9 27     0 3 15 63  Total Positivity and Neville Elimination – 3

• Gaussian Elimination    1 1 1 1         1 2 4 8         0 2 8 26     0 3 15 63  Total Positivity and Neville Elimination – 4

• Gaussian Elimination    1 1 1 1         0 1 3 7         0 2 8 26     0 3 15 63 

• In Matrix form:        1 1 1 1   1   1 1 1 1                     1 2 4 8   1 1   0 1 3 7          =            1 3 9 27   1 0 1   0 2 8 26         1 4 16 64   1 0 0 1   0 3 15 63  Total Positivity and Neville Elimination – 5

• Neville Elimination    1 1 1 1         1 2 4 8         1 3 9 27     1 4 16 64  Total Positivity and Neville Elimination – 6

• Neville Elimination    1 1 1 1         1 2 4 8         1 3 9 27     0 1 7 37  Total Positivity and Neville Elimination – 7

• Neville Elimination    1 1 1 1         1 2 4 8         0 1 5 19     0 1 7 37  Total Positivity and Neville Elimination – 8

• Neville Elimination    1 1 1 1         0 1 3 7         0 1 5 19     0 1 7 37  Total Positivity and Neville Elimination – 9

Neville Elimination – We can choose the pattern

   1 1 1 1         1 2 4 8      =    1 3 9 27     1 4 16 64 

     1   1 1 1 1               1   1 2 4 8               1   1 3 9 27       1   1 4 16 64  Total Positivity and Neville Elimination – 10

Neville Elimination – We can choose the pattern

   1 1 1 1         1 2 4 8      =    1 3 9 27     1 4 16 64 

     1   1 1 1 1               1   1 2 4 8               1   1 3 9 27       1 1   0 1 7 37  Total Positivity and Neville Elimination – 11

Neville Elimination – We can choose the pattern

   1 1 1 1         1 2 4 8      =    1 3 9 27     1 4 16 64 

       1   1   1 1 1 1                     1   1   1 2 4 8                     1   1 1   0 1 5 19         1 1   1   0 1 7 37  Total Positivity and Neville Elimination – 12

Neville Elimination – We can choose the pattern

   1 1 1 1         1 2 4 8      =    1 3 9 27     1 4 16 64 

       1   1   1 1 1 1                     1   1   1 2 4 8                     1   1 1   0 1 5 19         1 1   1 1   0 0 2 18  Total Positivity and Neville Elimination – 13

Neville Elimination – We can choose the pattern

   1 1 1 1         1 2 4 8      =    1 3 9 27     1 4 16 64 

         1   1   1   1 1 1 1                           1   1   1 1   0 1 3 7                           1   1 1   1   0 1 5 19           1 1   1 1   1   0 0 2 18  Total Positivity and Neville Elimination – 14

Neville Elimination – We can choose the pattern

   1 1 1 1         1 2 4 8      =    1 3 9 27     1 4 16 64 

         1   1   1   1 1 1 1                           1   1   1 1   0 1 3 7                           1   1 1   1 1   0 0 2 12           1 1   1 1   1   0 0 2 18  Total Positivity and Neville Elimination – 15

Neville Elimination – We can choose the pattern

   1 1 1 1         1 2 4 8      =    1 3 9 27     1 4 16 64 

         1   1   1   1 1 1 1                           1   1   1 1   0 1 3 7                           1   1 1   1 1   0 0 2 12           1 1   1 1   1 1   0 0 0 6  Total Positivity and Neville Elimination – 16

Neville Elimination – We can choose the pattern

   1 1 1 1         1 2 4 8      =    1 3 9 27     1 4 16 64 

           1   1   1   1   1 1 1 1                                 1   1   1 1   1   0 1 3 7                                 1   1 1   1 1   2   0 0 1 6             1 1   1 1   1 1   6   0 0 0 1  Total Positivity and Neville Elimination – 17

Neville Elimination – We can choose the pattern

   1 1 1 1         1 2 4 8      =    1 3 9 27     1 4 16 64 

             1   1   1   1   1 1 1 0   1                                       1   1   1 1   1   0 1 3 4   1                                       1   1 1   1 1   2   0 0 1 5   1 1               1 1   1 1   1 1   6   0 0 0 1   1  Total Positivity and Neville Elimination – 18

Neville Elimination – We can choose the pattern

   1 1 1 1         1 2 4 8      =    1 3 9 27     1 4 16 64 

               1   1   1   1   1 1   1   1                                             1   1   1 1   1   1 2   1 1   1                                             1   1 1   1 1   2   1 3   1 2   1 1                 1 1   1 1   1 1   6   1   1   1 

Notice: positive bidiagonal decomposition det(A(1 : k, i − k + 2 : i + 1)) det(A(1 : k − 1, i − k + 1 : i − 1)) U (k) = · . i,i+1 det(A(1 : k − 1, i − k + 2 : i)) det(A(1 : k, i − k + 1 : i))

Total Non-Negativity is preserved in Schur complementation Total Positivity and Neville Elimination – 19

• Neville Elimination yield Bidiagonal Decomposition of Any Totally Positive Matrix  + + + +       + + + +  BD   =    + + + +   + + + + 

 1   1   1   +   1 +   1   1                               1   1   + 1   +   1 +   1 +   1                               1   + 1   + 1   +   1 +   1 +   1 +   + 1   + 1   + 1   +   1   1   1  • n2 entries of factors intrinsically parametrize all TP matrices • For numerical computation: – Discard A Instead: Use BD(A) above – Let A → A0 be a linear transformation ∗ Subtract a row from another, apply Givens rot, etc. – We will compute BD(A0) from BD(A) without losing relative accuracy Accurate Linear Algebra with BD(A)

• With BD(A) we can do accurately – Positive diagonal scaling – trivial – Subtract a row/column from next to create a zero – Multiply by a positive bidiagonal matrix • All linear algebra can be “assembled” from above Subtract a row/column from next to create a zero – 1

• Equivalent to setting an entry in BD(A) to 0 and doing no arithmetic

 + + + +       + + + +    =    + + + +   + + + + 

 1   1   1   +   1 +   1   1                               1   1   + 1   +   1 +   1 +   1                               1   + 1   + 1   +   1 +   1 +   1 +   + 1   + 1   + 1   +   1   1   1  | {z }  + + + +       + + + +       + + + +   0 + + +  Subtract a row/column from next to create a zero – 2

• Equivalent to setting an entry in BD(A) to 0 and doing no arithmetic  + + + +       + + + +    =    + + + +   + + + + 

 1   1   1   +   1 +   1   1                               1   1   + 1   +   1 +   1 +   1                               1   + 1   + 1   +   1 +   1 +   1 +   + 1   + 1   + 1   +   1   1   1  | {z }  + + + +       + + + +       + + + +   0 + + +   + + + +       + + + +    =    + + + +   0 + + + 

 1   1   1   +   1 +   1   1                               1   1   + 1   +   1 +   1 +   1                               1   + 1   + 1   +   1 +   1 +   1 +   0 1   + 1   + 1   +   1   1   1  Accurate TP×BIDIAGONAL

  1             1     BD BD(A) ×   =?   + 1       1  “Commuting” of positive bidiagonals

• Assume x, y, z > 0, bi ≥ 0. Then       1 1       0 1 b1     1 b        1     1   1           0   1 b2       1 b       0   2     y   y     .. ..       .. ..   . .    =  0 0   . .     x z   x z           0   1 b       1 b   n−1       n−1     1   1    1     1  1   1  where x0 = x 0 y = y + bix z0 = 1/y0 0 bi−1 = bi−1y 0 bi = biz/y1 • No loss of relative accuracy • “Commute accurately”. “Commuting” of positive bidiagonals - 2

• Assume x > 0, bi ≥ 0. Then       1 1       1     1            1   1           0   b1 1       b 1         1   . .   1   1   . .   . .       .. ..   . .    =        x 1   1           0   b 1       b 1   n−2     0   n−2     1   x 1    b 1     b0 1 n−1  1   1  n−1 where

0 x = xbi+1/(bi + x) 0 bi = bi + x 0 0 bi+1 = xbi/bi • No loss of relative accuracy Applying similarity to bidiagonal form – 1

 1   1   1   +   1 +   1   1   1                                   1   1   + 1   +   1 +   1 +   1   1                                   1   + 1   + 1   +   1 +   1 +   1 +   1   + 1   + 1   + 1   +   1   1   1   + 1  Applying similarity to bidiagonal form – 2

 1   1   1   +   1 +   1   1   1                                   1   1   + 1   +   1 +   1 +   1   1                                   1   + 1   + 1   +   1 +   1 +   y   1 +   + 1   + 1   + 1   +   1   1   + z   1  Applying similarity to bidiagonal form – 3

 1   1   1   +   1 +   1   1   1                                   1   1   + 1   +   1 +   1   1 +   1                             0       1   + 1   + 1   +   1 +   y   1 +   1 +   + 1   + 1   + 1   +   1   + z0   1   1  Applying similarity to bidiagonal form – 4

 1   1   1   +   1   1 +   1   1                                   1   1   + 1   +   1   1 +   1 +   1                           00         1   + 1   + 1   +   y   1 +   1 +   1 +   + 1   + 1   + 1   +   + z00   1   1   1  Applying similarity to bidiagonal form – 5

 1   1   1   1   +   1 +   1   1                                   1   1   + 1   1   +   1 +   1 +   1                                   1   + 1   + 1   1   +   1 +   1 +   1 +   + 1   + 1   + 1   + 1   +   1   1   1  Applying similarity to bidiagonal form – 6

 1   1   1   +   1 +   1   1                               1   1   + 1   +   1 +   1 +   1                               1   + 1   + 1   +   1 +   1 +   1 +   + 1   + 1   + 1   +   1   1   1 

DONE! Accurate Matrix Computations with TP matrices

• BD(TP × TP) – suffices BD(TP × BIDIAGONAL) – done • BD(diag(±1) · (TP)−1 · diag(±1)) – OK, still a product of positive bidiagonals • BD(SchurComplement) – also OK, but barely more complicated • BD(Givens · TP )

       c s   + +   1 0    =      −s c   0 +   − 1  • SVD is then OK: TP→BIDIAGONAL→SVD – no loss of accuracy • Eigenvalues: First reduce to tridiagonal Reduction of a nonsymmetric matrix to tridiagonal form – 1

   + + + +         + + + +         + + + +     + + + +  Reduction of a nonsymmetric matrix to tridiagonal form – 2

     1   + + + +               1   + + + +               1   + + + +       − 1   + + + + 

   + + + +         + + + +    =      + + + +     0 + + +  Reduction of a nonsymmetric matrix to tridiagonal form – 3

       1   + + + +   1                     1   + + + +   1                     1   + + + +   1         − 1   + + + +   + 1 

     + + + +   1               + + + +   1      =          + + + +   1       0 + + +   + 1  Reduction of a nonsymmetric matrix to tridiagonal form – 4

       1   + + + +   1                     1   + + + +   1                     1   + + + +   1         − 1   + + + +   + 1 

     + + + +   1               + + + +   1      =          + + + +   1       0 + + +   + 1 

   + + + +         + + + +    =      + + + +     0 + + +  Reduction of a nonsymmetric matrix to tridiagonal form – 5

   + + + +         + + + +         + + + +     0 + + +  Reduction of a nonsymmetric matrix to tridiagonal form – 6

     + + + +   1               + + + +   1               + + + +   1 −       0 + + +   1 

   + + + 0         + + + +    =      + + + +     0 + + +  Reduction of a nonsymmetric matrix to tridiagonal form – 7

       1   + + + +   1                     1   + + + +   1                     1 +   + + + +   1 −         1   0 + + +   1 

     1   + + + 0               1   + + + +      =          1 +   + + + +       1   0 + + +  Reduction of a nonsymmetric matrix to tridiagonal form – 8

       1   + + + +   1                     1   + + + +   1                     1 +   + + + +   1 −         1   0 + + +   1 

     1   + + + 0               1   + + + +      =          1 +   + + + +       1   0 + + + 

   + + + 0         + + + +    =      + + + +     0 + + +  Reduction of a nonsymmetric matrix to tridiagonal form – 9

       1   + + + +   1                     1   + + + +   1                     1 +   + + + +   1 −         1   0 + + +   1 

     1   + + + 0               1   + + + +      =          1 +   + + + +       1   0 + + + 

   + + + 0         + + + +    =      0 + + +     0 + + +  Reduction of a nonsymmetric matrix to tridiagonal form – 10

       1   + + + +   1                     1   + + + +   1                     1 +   + + + +   1 −         1   0 + + +   1 

     1   + + + 0               1   + + + +      =          1 +   + + + +       1   0 + + + 

   + + 0 0         + + + +    =      0 + + +     0 + + +  Reduction of a nonsymmetric matrix to tridiagonal form – 11

       1   + + + +   1                     1   + + + +   1                     1 +   + + + +   1 −         1   0 + + +   1 

     1   + + + 0               1   + + + +      =          1 +   + + + +       1   0 + + + 

   + + 0 0         + + + +    =      0 + + +     0 0 + +  Reduction of a nonsymmetric matrix to tridiagonal form – 12

       1   + + + +   1                     1   + + + +   1                     1 +   + + + +   1 −         1   0 + + +   1 

     1   + + + 0               1   + + + +      =          1 +   + + + +       1   0 + + + 

   + + 0 0         + + + 0    =      0 + + +     0 0 + +  Accurate Eigenvalues of Tridiagonals The eigenvalues of

       1   d1   1 c1                     b1 1   d2   1 c2               .. ..   ..   .. ..   . .   .   . .                     bn−2 1   dn−1   1 cn−1              bn−1 1 dn 1 are the squares of the singular values of  √ √   d1 d1b1c1     √ √     d d b c   2 2 2 2     . .   .. ..     √ √       dn−1 dn−1bn−1cn−1   √    dn • Thus the eigenvalue problem of a TP matrix is reduced to the SVD problem for a bidiagonal matrix • Forward stable • Total cost = 8n3. Computing Accurate BD(A)

• Bidiagonal decompositions intrinsic for all TP matrices • Often accurate formulas: – Vandermonde: i−1 i−1 x − x Y Y i+1 j+1 (xi − xj), , xi+n−k j=1 j=i−k+1 xi − xj – Cauchy:

n−1 (x − x )(y − y ) x + y i−1 x − x i−n+k−1 x + y Y n k n k , n−k i−n+k+1 Y i+1 l+1 · Y i l k=1 (xn + yk)(yn + xk) xi + yi−n+k+1 l=n−k xi − xl l=1 xi+1 + yl

y + x i−1 y − y i−n+k−1 y + x n−k i−n+k+1 Y i+1 l+1 · Y i l , yi + xi−n+k+1 l=n−k yi − yl l=1 yi+1 + xl • Equivalent to Bj¨orck-Pereyra Methods • Can be computed iff initial minors can be computed accurately det(A(1 : k, i − k + 2 : i + 1)) det(A(1 : k − 1, i − k + 1 : i − 1)) U (k) = · . i,i+1 det(A(1 : k − 1, i − k + 2 : i)) det(A(1 : k, i − k + 1 : i))

initial minors = contiguous, include the first row or column Conclusions

• O(n3) algorithms for the eigenvalues and the SVD of Totally Positive Matrices to high relative accuracy • Accurate linear algebra with TP matrices closed under same op- erations as TP • Applies to: – Oscillatory – Totally Nonnegative Matrices – Sign Regular (Inverses of TP matrices) • Open problem: If we perturb an entry in (PositiveBidiagonal) · (PositiveBidiagonal) How much do the signular vectors change? Singular values provably change by at most . • This talk and Matlab software: math.mit.edu/~plamen