Index

A-optimality, 439 angle between matrices, 168 absolute error, 484, 494, 526 angle between vectors, 37, 229, 357 ACM Transactions on Mathematical ANSI (standards), 475, 562, 563 Software, 552, 617 Applied Statistics algorithms, 617 adj(·), 69 approximation and estimation, 431 adjacency , 331–334, 390 approximation of a matrix, 174, 257, Exercise 8.20:, 396 339, 437, 533 augmented, 390 approximation of a vector, 41–43 adjoint (see also conjugate transpose), arithmetic mean, 35, 37 59 Arnoldi method, 318 adjoint, classical (see also adjugate), 69 artificial ill-conditioning, 269 adjugate, 69, 598 ASCII code, 460 adjugate and inverse, 118 association matrix, 328–338, 357, 365, affine group, 115, 179 366, 369–371 affine space, 43 , 331–332, 390 affine transformation, 228 connectivity matrix, 331–332, 390 Aitken’s integral, 217 dissimilarity matrix, 369 AL(·) (affine group), 115 , 369 algebraic multiplicity, 144 , 331–332, 390 similarity matrix, 369 algorithm, 264, 499–513 ATLAS (Automatically Tuned Linear batch, 512 Algebra Software), 555 definition, 509 augmented adjacency matrix, 390 direct, 264 augmented connectivity matrix, 390 divide and conquer, 506 Automatically Tuned greedy, 506 Software (ATLAS), 555 iterative, 264, 508–511, 564 autoregressive process, 447–450 reverse communication, 564 axpy, 12, 50, 83, 554, 555 online, 512 axpy elementary operator matrix, 83 out-of-core, 512 real-time, 512 back substitution, 274, 406 algorithmically singular, 121 backward error analysis, 495, 500 Anaconda, 553, 556, 569 Banach space, 33 angle(·, ·), 37 Banachiewicz factorization, 255 632 Index banded matrix, 58 Cartesian geometry, 35, 73 inverse, 121 catastrophic cancellation, 486 Bartlett decomposition, 346 , 389 base, 467 Cauchy-Schwarz inequality, 24, 98 base point, 466 Cauchy-Schwarz inequality for matrices, , 21–23 98, 178 Exercise 2.6:, 52 Cayley multiplication, 75, 94 orthonormal, 40–41 Cayley-Hamilton theorem, 138 batch algorithm, 512 CDF (Common Data Format), 463 Bauer-Fike theorem, 307 centered matrix, 288, 363 Beowulf (cluster computing), 559 centered vector, 49 bias, in exponent of floating-point chaining of operations, 485 number, 468 character data, 461 big endian, 480 character string, 461 big integer, 466, 492 characteristic equation, 138 big O (order), 497, 503, 591 characteristic polynomial, 138 big omega (order), 497 characteristic value (see also eigenvalue), bilinear form, 91, 134 135 bit, 460 characteristic vector (see also eigenvec- bitmap, 461 tor), 135 BLACS (software), 557, 558 chasing, 317 BLAS (software), 553–556 Chebyshev norm, 28 CUDA, 560 chi-squared distribution, 400 PBLAS, 558 noncentral, 400 PLASMA, 559 PDF, equation (9.3), 400 PSBLAS, 558 Cholesky decomposition, 253–256, 345, block , 62 436 determinant of, 70 computing, 558, 574 inverse of, 121 root-free, 255 multiplication, 78 , 383 BMvN distribution, 219, 548 classification, 389 Bolzano-Weierstrass theorem for cluster analysis, 389 orthogonal matrices, 133 cluster computing, 559 , 390 coarray (Fortran construct), 563 Box M statistic, 368 Cochran’s theorem, 353–356, 401 bra·ket notation, 24 cofactor, 68, 598 byte, 460 Collected Algorithms of the ACM (CALGO), 617 C (programming language), 474, 489, collinearity, 265, 406, 430 568–569 column rank, 99 C++ (programming language), 475, column space, 55, 89, 104, 105 568–569 column-major, 522, 539, 545 CALGO (Collected Algorithms of the column-sum norm, 165 ACM), 617 Common Data Format (CDF), 463 cancellation error, 487, 500 , 139, 305 canonical form, equivalent, 110 compatible linear systems, 106 canonical form, similar, 149 compensated summation, 486 canonical singular value factorization, complementary , 356 162 complementary vector spaces, 19 Index 633 complete graph, 329 convergence of powers of a matrix, 171, complete pivoting, 275 375 complete space, 33 convergence rate, 509 completing the Gramian, 177 convex combination, 12 complex data type, 491 convex cone, 44–46, 346, 349, 371 complex vectors/matrices, 33, 132, 387 convex function, 26, 197 Conda, 553 convex optimization, 346 condition (problem or data), 499 convex set, 12 condition number, 265, 270, 290, 426, convexity, 26 427, 499, 502, 523, 533 coordinate, 4 computing the number, 533, 574 Cor(·, ·), 51 inverse of matrix, 266, 270 correlation, 51, 90 nonfull rank matrices, 290 correlation matrix, 365, 422 nonsquare matrices, 290 Exercise 8.8:, 394 sample standard deviation, 502 positive definite approximation, 436 conditional inverse, 128 pseudo-correlation matrix, 437 cone, 43–46, 327 sample, 422 Exercise 2.18:, 53 cost matrix, 369 ·, · convex cone, 44, 346 Cov( ), 50 covariance, 50 of nonnegative definite matrices, 346 , see variance- of nonnegative matrices, 371 covariance matrix of positive definite matrices, 349 CRAN (Comprehensive R Archive of positive matrices, 371 Network), 552, 576 , 381 cross product of vectors, 47 configuration matrix, 369 Exercise 2.19:, 54 conjugate gradient method, 279–283 cross products matrix, 256, 358 preconditioning, 282 cross products, computing sum of conjugate norm, 93 Exercise 10.18c:, 519 conjugate transpose, 59, 132 Crout method, 245 conjugate vectors, 93, 134 cuBLAS, 560 connected vertices, 329, 334 CUDA, 559 connectivity matrix, 331–334, 390 curl, 192 Exercise 8.20:, 396 curse of dimensionality, 511 augmented, 390 cuSPARSE, 560 consistency property of matrix norms, 164 D-optimality, 439–440, 533 consistency test, 527, 551 daxpy, 12 consistent system of equations, 105, decomposable matrix, 373 272, 277 decomposition, see also factorization of constrained least squares, equality a matrix constraints, 413 additive, 353, 354 Exercise 9.4d:, 451 Bartlett decomposition, 346 continuous function, 186 multiplicative, 108 contrast, 410 factorization, convergence criterion, 508 257, 337 convergence of a sequence of matrices, singular value decomposition, 134, 152, 171 161–164, 320, 336, 425, 532 convergence of a sequence of vectors, 32 spectral decomposition, 155 634 Index defective (deficient) matrix, 149, 150 dimension of vector space, 14 deficient (defective) matrix, 149, 150 dimension reduction, 20, 356, 425 deflation, 308–310 direct method for solving linear systems, degrees of freedom, 360, 362, 407, 430 272–277 del, 192 direct product (of matrices), 95 derivative with respect to a matrix, 195 direct product (of sets), 5 derivative with respect to a vector, direct product (of vector spaces), 20 189–195 basis for, 23 det(·), 66 direct sum decomposition of a vector determinant, 66–74 space, 19 as criterion for optimal design, 439 direct sum of matrices, 62 computing, 533 direct sum of vector spaces, 18–20, 64 derivative of, 195 basis for, 22 Jacobian, 217 direct sum decomposition, 19 of block diagonal matrix, 70 directed dissimilarity matrix, 369 of Cayley product, 88 direction cosines, 38, 231 of diagonal matrix, 70 discrete Fourier transform, 384 of elementary operator matrix, 86 discrete Legendre polynomials, 380 of inverse, 117 discretization error, 498, 509 of Kronecker product, 96 dissimilarity matrix, 369 of nonnegative definite matrix, 345 distance, 32 of partitioned matrix, 70, 122 between matrices, 175 of , 87 between vectors, 32 of positive definite matrix, 347 distance matrix, 369 of transpose, 70 distributed computing, 463, 508, 544 of , 70 distributed linear algebra machine, 558 relation to eigenvalues, 141 distribution vector, 377 relation to geometric volume, 73, 217 div, 192 diag(·), 56, 60, 596 divergence, 192 with matrix arguments, 62 divide and conquer, 506 diagonal element, 56 document-term matrix, 336 diagonal expansion, 73 dominant eigenvalue, 142 diagonal factorization, 148, 152 Doolittle method, 245 diagonal matrix, 57 dot product of matrices, 97 determinant of, 70 dot product of vectors, 23, 91 inverse of, 120 double cone, 43 multiplication, 78 double , 472, 480 , 148–152, 306, doubly , 377 343 Drazin inverse, 129–130 orthogonally, 154 dual cone, 44 unitarily, 147, 343, 387 diagonally dominant matrix, 57, 62, Epq, E(π), Ep(a), Epq(a) (elementary 100, 348 operator matrices), 84 differential, 188 E(·) (expectation operator), 216 differentiation of vectors and matrices, E-optimality, 439 183–220 echelon form, 111 digraph, 332 edge of a graph, 329 of a matrix, 333 EDP (exact dot product), 493 dim(·), 14 effective degrees of freedom, 362, 430 Index 635 efficiency, computational, 503–508 Euclidean distance matrix, 369 eigenpair, 134 Euclidean matrix norm (see also eigenspace, 144 Frobenius norm), 167 eigenvalue, 134–164, 166, 305–322 Euclidean vector norm, 27 computing, 306–318, 558, 574 Euler’s constant Exercise 10.2:, 516 Jacobi method, 313–316 Euler’s integral, 593 Krylov methods, 318 Euler’s rotation theorem, 231 power method, 311–313 exact computations, 493 QR method, 316–318 exact dot product (EDP), 493 of a graph, 391 exception, in computer operations, 483, of a polynomial Exercise 3.26:, 181 487 relation to singular value, 163 expectation, 212–220 upper bound on, 142, 145, 306 exponent, 467 eigenvector, 134–164, 305–322 exponential order, 503 left eigenvector, 135, 158 exponential, matrix, 153, 184 eigenvectors, of, extended precision, 472 143 extrapolation, 509 EISPACK, 556 elementary operation, 79 factorization of a matrix, 108, 112, 147, elementary operator matrix, 79–87, 101, 148, 161, 225–227, 239–259, 272, 242, 273 274 eigenvalues, 137 Banachiewicz factorization, 255 elliptic metric, 94 Bartlett decomposition, 346 elliptic norm, 93 canonical singular value factorization, endian, 480 162 equivalence of norms, 29, 33, 170 Cholesky factorization, 253–256 equivalence relation, 444 diagonal factorization, 148 equivalent canonical factorization, 112 equivalent canonical factorization, equivalent canonical form, 110, 112 112 equivalent matrices, 110 full rank factorization, 108, 112 error bound, 496 Gaussian elimination, 272 error in computations LQ factorization, 247 cancellation, 487, 500 LU or LDU factorization, 240–246 error-free computations, 493 nonnegative matrix factorization, measures of, 484, 494–497, 526 257, 337 rounding, 487, 495 orthogonally diagonal factorization, Exercise 10.10:, 517 147 error of approximation, 498 QL factorization, 247 discretization, 498 QR factorization, 246–252 truncation, 498 root-free Cholesky, 255 error, measures of, 272, 484, 494–497, RQ factorization, 247 526 Schur factorization, 147 error-free computations, 493 singular value factorization, 161–164, errors-in-variables, 405 320, 336, 425, 532 essentially disjoint vector spaces, 15, 64 square root factorization, 160 estimable combinations of parameters, unitarily diagonal factorization, 147 409 fan-in algorithm, 485, 506 estimation and approximation, 431 fast Fourier transform (FFT), 386 Euclidean distance, 32, 369 fast Givens rotation, 239, 525 636 Index

fill-in, 259, 526 geometric multiplicity, 144 , 205 geometry, 35, 73, 227, 231 fixed-point representation, 465 Gershgorin disks, 145 flat, 43 GitHub, 539 floating-point representation, 466 Exercise 12.3:, 581 FLOP, or flop, 505 Givens transformation (rotation), FLOPS, or flops, 505 236–239, 317 Fortran, 475–478, 505, 546, 566–568 QR factorization, 251 Fourier coefficient, 41, 42, 99, 157, 163, GL(·) (general linear group), 114 169 GMP (software library), 466, 491, 492 Fourier expansion, 36, 41, 98, 157, 163, Exercise 10.5:, 516 169 GMRES, 282 Fourier matrix, 384 GNU Scientific Library (GSL), 556 Frobenius norm, 167–169, 171, 175, 314, GPU (graphical processing unit), 559 340, 369 graceful underflow, 470 Frobenius p norm, 169 gradient, 189 full precision, 480 projected gradient, 207 full rank, 100, 101, 104, 111–113 reduced gradient, 207 full rank factorization, 108 gradient descent, 198, 199 , 112 gradient of a function, 190, 191 full rank partitioning, 104, 122 gradual underflow, 470, 487 Gram-Schmidt transformation, 39, 40, g1 inverse, 128, 130 524 g2 inverse, 128 linear least squares, 289 g4 inverse (see also Moore-Penrose QR factorization, 252 inverse), 128 Gramian matrix, 115, 117, 256, 289, gamma function, 220, 593 358–359 GAMS (Guide to Available Mathemati- completing the Gramian, 177 cal Software), 539 graph of a matrix, 332 Gauss (software), 570 , 8, 329–336, 389 Gauss-Markov theorem, 411 graphical processing unit (GPU), 559 Gauss-Newton method, 203 greedy algorithm, 506 Gauss-Seidel method, 277 group, 114, 133 Gaussian elimination, 84, 272, 317 GSL (GNU Scientific Library), 556 Gaussian matrix, 84, 241 guard digit, 485 gemm (general matrix-matrix), 556 gemv (general matrix-vector), 556 Haar distribution, 219, 549 general linear group, 114, 133 Exercise 4.10:, 221 generalized eigenvalue, 160, 319 Exercise 8.8:, 394 , 124–125, 127–131, Haar invariant measure, 220 249, 359 , 380 relation to QR factorization, 249 Hadamard multiplication, 94 generalized least squares, 414 Hadamard’s inequality Exercise 5.5:, generalized least squares with equality 260, 606 constraints Exercise 9.4d:, 451 Hadoop, 464, 544 generalized variance, 366 Hadoop Distributed File System generating set, 14, 21 (HDFS), 464, 514 of a cone, 44 half precision, 480 generation of random numbers, 440 , 388 Index 637

Hankel norm, 389 ill-posed problem, 121 hat matrix, 360, 408 image data, 461 HDF, HDF5 (Hierarchical Data IMSL Libraries, 556, 560–562 Format), 463 incidence matrix, 331–334, 390 HDFS (Hadoop Distributed File incomplete data, 435–438 System), 464, 514 incomplete factorization, 258, 526 Helmert matrix, 378, 410 independence, linear, see linear Hemes formula, 286, 415 independence Hermite form, 111 independent vertices, 390 , 56, 60 index-index-value (sparse matrices), 548 , 59, 316 induced matrix norm, 165 , 193 infinity, floating-point representation, projected Hessian, 207 473, 487, 488 reduced Hessian, 207 infix operator, 489 Hessian of a function, 193 inner product, 23, 245 hidden bit, 468 inner product of matrices, 96–99 Hierarchical Data Format (HDF), 463 inner product space, 24 high-performance computing, 507 inner pseudoinverse, 128 , 548 integer representation, 465 Hilbert space, 33, 168 integration and expectation, 212–220 Hilbert-Schmidt norm (see also integration of vectors and matrices, 213 Frobenius norm), 167 Intel Math Kernel Library (MKL), 555, Hoffman-Wielandt theorem, 339 578 H¨older norm, 27 intersection graph, 334 H¨older’s inequality Exercise 2.11a:, 52 intersection of vector spaces, 18 , 57, 369 interval arithmetic, 492, 493 homogeneous coordinates, 232 invariance property, 227 in graphics applications, Exercise 5.2:, invariant distribution, 445 259 invariant vector (eigenvector), 135 homogeneous system of equations, 43, inverse of a matrix, 107 123 determinant of, 117 Horner’s method, 512 Drazin inverse, 129–130 Householder transformation (reflection), generalized inverse, 124–125, 127–131 233–236, 250, 317 Drazin inverse, 129–130 hyperplane, 43 Moore-Penrose inverse, 127–129 hypothesis testing, 408 pseudoinverse, 128 Kronecker product, 118 , 350–357 left inverse, 108 , 60, 76 Moore-Penrose inverse, 127–129 IDL (software), 6, 570 partitioned matrix, 122 IEC standards, 464 products or sums of matrices, 118 IEEE standards, 464, 487 pseudoinverse, 128 Standard 754, 472, 480, 487, 494 right inverse, 108 Standard P1788, 493 transpose, 107 IFIP Working Group 2.5, 460, 494 triangular matrix, 121 ill-conditioned (problem or data), 264, inverse of a vector, 31 426, 499, 523 IRLS (iteratively reweighted least artificial, 269 squares), 297 stiff data, 502 irreducible Markov chain, 444 638 Index irreducible matrix, 311, 335–336, LAPACK, 276, 534, 556, 558 372–377, 445 LAPACK95, 556 is.na, 473 Laplace expansion, 68 isnan, is.nan, 473 Laplace operator (∇2), 599 ISO (standards), 475, 562, 563 Laplace operator (∇2), 192 isometric matrix, 167 , 391 isometric transformation, 227 lasso regression, 430 isotropic transformation, 228 latent root (see also eigenvalue), 135 iterative method, 277, 284, 305, LAV (least absolute values), 295 508–511, 525 LDU factorization, 240–246 for solving linear systems, 277–283 leading principal submatrix, 62, 348 iterative refinement, 284 least absolute values, 295 iteratively reweighted least squares, 297 least squares, 200–204, 256, 286–294 constrained, 206–211 Jacobi method for eigenvalues, 313–316 nonlinear, 202–204 Jacobi transformation (rotation), 236 least squares regression, 201 Jacobian, 191, 217 left eigenvector, 135, 158 Jordan block, 77, 139 left inverse, 108 Jordan decomposition, 151 length of a vector, 4, 27, 31 Jordan form, 77, 111 Leslie matrix, 378, 446 of , 77 Exercise 8.10:, 395 Exercise 9.22:, 455 Kalman filter, 502 Levenberg-Marquardt method, 204 Kantorovich inequality, 350 leverage, 408 Karush-Kuhn-Tucker conditions, 210 Exercise 9.6:, 451 kind (for data types), 476 life table, 447 Kronecker multiplication, 94–96 likelihood function, 204 inverse, 118 line, 43 properties, 95 linear convergence, 509 symmetric matrices, 96, 156 linear estimator, 409 diagonalization, 156 linear independence, 12, 99 Kronecker structure, 219, 419 linear independence of eigenvectors, 143 Krylov method, 281, 318 linear programming, 346 Krylov space, 281 linear regression, 401–422, 426–431 Kuhn-Tucker conditions, 210 variable selection, 427 Kulisch accumulator, 493 LINPACK, 276, 533, 556 Kullback-Leibler divergence, 175 Lisp-Stat (software), 570 little endian, 480 L1, L2, and L∞ norms little o (order), 497, 591 of a matrix, 165 little omega (order), 497 of a symmetric matrix, 167 log order, 503 of a vector, 27 log-likelihood function, 205 relations among, 170 Longley data Exercise 9.10:, 453 L2 norm of a matrix (see also spectral loop unrolling, 565 norm), 166 Lorentz cone, 44 Lagrange multiplier, 208, 414 lower triangular matrix, 57 Exercise 9.4a:, 450 Lp norm Lagrangian function, 208 of a matrix, 165 Lanczos method, 318 of a vector, 27–28, 186 Index 639

LQ factorization, 247 MCMC (Markov chain Monte Carlo), LR method, 306 445 LU factorization, 240–246 mean, 35, 37 computing, 558, 574 mean vector, 35 message passing, 557 M-matrix, 393 Message Passing Library, 557 MACHAR, 478 metric, 32, 175 Exercise 10.3(d)i:, 516 metric space, 32 machine epsilon, 470 Microsoft R Open, 578 Mahalanobis distance, 94, 365 MIL-STD-1753 standard, 478 Manhattan norm, 27 Minkowski inequality, 27 manifold of a matrix, 55 Exercise 2.11b:, 52 Maple (software), 491, 570 Minkowski norm, 27 MapReduce, 463, 513, 531, 544 , 67, 597 Markov chain, 443–445 missing data, 435–438, 571 Markov chain Monte Carlo (MCMC), representation of, 462, 473 445 MKL (Intel Math Kernel Library), 555, Mathematica (software), 491, 570 578 Matlab (software), 546, 578–580 mobile Jacobi scheme, 315 matrix, 5 modified Cholesky decomposition, 436 matrix derivative, 183–220 “modified” Gauss-Newton, 203 , 153, 184 “modified” Gram-Schmidt (see also matrix factorization, 108, 112, 147, 148, Gram-Schmidt transformation), 161, 225–227, 239–259, 272, 274 40 matrix function, 152 Moore-Penrose inverse, 127–129, 248, matrix gradient, 191 249, 291 matrix inverse, 107 relation to QR factorization, 249 , 75–99, 528 MPI (message passing interface), 557, Cayley, 75, 94 559 CUDA, 560 MPL (Message Passing Library), 557 Hadamard, 94 multicollinearity, 265, 406 inner product, 96–99 multigrid method, 283 Kronecker, 94–96 multiple precision, 466, 491 MapReduce, 531 multiplicity of an eigenvalue, 144 Strassen algorithm, 529–531 multivariate gamma function, 220 matrix norm, 164–171 multivariate linear regression, 418–422 orthogonally invariant, 164 multivariate normal distribution, matrix normal distribution, 218 217–219, 399, 441 matrix of type 2, 58, 383 singular, 217, 432 matrix pencil, 161 multivariate random variable, 215–220 matrix polynomial, 78 Exercise 3.26:, 181 N (·), 126 matrix random variable, 218–220 NA (“Not Available”), 462, 473, 571 matrix storage mode, 546–548 nabla (∇), 190, 191 Matrix Template Library, 569 Nag Libraries, 556 max norm, 28 NaN (“Not-a-Number”), 473, 488 maximal linearly independent subset, NetCDF, 463 12 netlib, xiv, 617 maximum likelihood, 204–206 network, 329–336, 391 640 Index

Newton’s method, 198 one vector, 16, 34 nilpotent matrix, 77, 174 online algorithm, 512 NMF (nonnegative matrix factoriza- online processing, 512 tion), 257, 337 open-source, 538 noncentral chi-squared distribution, 400 OpenMP, 557, 559 PDF, equation (9.3), 400 operator matrix, 80, 273 noncentral Wishart distribution operator norm, 165 Exercise 4.12:, 222 optimal design, 438–440 nonlinear regression, 201 optimization of vector/matrix functions, nonnegative definite matrix, 91, 159, 196–212 253, 344–350 constrained, 206–211 summary of properties, 344–345 least squares, 200–204, 206–211 nonnegative matrix, 258, 369 order of a graph, 329 nonnegative matrix factorization, 257, order of a vector, 4 337 order of a vector space, 15 nonsingular matrix, 100, 111 order of computations, 503 norm, 25–30 order of convergence, 497 convexity, 26 order of error, 497 equivalence of norms, 29, 33, 170 ordinal relations among matrices, 92, of a matrix, 164–171 348 orthogonally invariant, 164 ordinal relations among vectors, 16 of a vector, 27–31 orthogonal array, 380 weighted, 28, 93 orthogonal basis, 40–41 normal distribution, 217–219 orthogonal complement, 34, 126, 131 matrix, 218 orthogonal distance regression, 299–302, multivariate, 217–219 405 normal equations, 256, 289, 404, 420 orthogonal group, 133, 220 , 342 orthogonal matrices, binary relation- Exercise 8.1:, 394 ship, 98 circulant matrix, 384 , 131–134, 228 normal vector, 34 orthogonal residuals, 299–302, 405 normalized floating-point numbers, 468 orthogonal transformation, 228 normalized generalized inverse (see also orthogonal vector spaces, 34, 131 Moore-Penrose inverse), 128 orthogonal vectors, 33 normalized vector, 31 Exercise 2.6:, 52 normed space, 25 orthogonalization (Gram-Schmidt not-a-number (“NaN”), 473 transformations), 38, 252, 524 NP-complete problem, 504 orthogonally diagonalizable, 147, 154, nuclear norm, 169 338, 343, 423 null space, 126, 127, 144 orthogonally invariant norm, 164, 167, nullity, 126 168 numpy, 556, 569 orthogonally similar, 146, 154, 164, 168, Nvidia, 559 269, 343 orthonormal vectors, 33 O(·), 497, 503, 591 out-of-core algorithm, 512 o(·), 497, 591 outer product, 90, 245 oblique projection, 356 Exercise 3.14:, 179 Octave (software), 578 outer product for matrix multiplication, OLS (ordinary least squares), 288 529 Index 641 outer pseudoinverse, 128, 129 positive matrix, 258, 369 outer/inner products matrix, 357 positive semidefinite matrix, 91 overdetermined linear system, 124, 255, positive stable, 159, 393 286 power method for eigenvalues, 311–313 overfitting, 298, 429 precision, 472–480, 491 overflow, in computer operations, 483, arbitrary, 466 487 double, 472, 480 Overleaf, 539 extended, 472 overloading, 11, 63, 165, 479, 489 half precision, 480 infinite, 466 p-inverse (see also Moore-Penrose multiple, 466, 491 inverse), 128 single, 472, 480 paging, 565 preconditioning, 282, 310, 525 parallel processing, 507, 508, 528, 530, for eigenvalue computations, 310 544, 557 in the conjugate gradient method, parallelogram equality, 27 282 parallelotope, 74 primitive Markov chain, 445 Parseval’s identity, 41, 169 primitive matrix, 375, 445 partial ordering, 16, 92, 348 principal axis, 36 Exercise 8.2a:, 394 principal components, 422–426 partial pivoting, 275 principal components regression, 428 partitioned matrix, 61, 79, 131 principal diagonal, 56 determinant, 70, 122 principal minor, 72, 104, 598 sum of squares, 361, 413, 420 principal submatrix, 62, 104, 241, 344, partitioned matrix, inverse, 122, 131 347 partitioning sum of squares, 361, 413, leading, 62, 348 420 probabilistic error bound, 497 PBLAS (parallel BLAS), 558 programming model, 463, 544 pencil, 161 projected gradient, 207 permutation, 66 projected Hessian, 207 permutation matrix, 80, 86, 273, 377 projection (of a vector), 20, 36 Perron root, 371, 374 projection matrix, 356–357, 408 Perron theorem, 371 projective transformation, 228 Perron vector, 371, 375, 445 proper value (see also eigenvalue), 135 Perron-Frobenius theorem, 374 PSBLAS (parallel sparse BLAS), 558 pivoting, 84, 244, 249, 275 pseudo-correlation matrix, 437 PLASMA, 559 pseudoinverse (see also Moore-Penrose polar cone, 45 inverse), 128 polynomial in a matrix, 78 PV-Wave (software), 6, 570 Exercise 3.26:, 181 Pythagorean theorem, 27 polynomial order, 503 Python, 478, 546, 569, 570 polynomial regression, 379 polynomial, evaluation of, 512 Q-convergence, 509 pooled variance-covariance matrix, 368 QL factorization, 247 population model, 445 QR factorization, 246–252 portability, 481, 494, 540 and a generalized inverse, 249 positive definite matrix, 91, 101, computing, 558, 574 159–160, 253, 346–350, 422 matrix rank, 250 summary of properties, 346–348 skinny, 247 642 Index

QR method for eigenvalues, 316–318 Rayleigh quotient, 90, 157, 209, 392 quadratic convergence, 509 Rcpp, 577 quadratic form, 91, 93 RcppBlaze, 559 quasi-Newton method, 199 RCR (Replicated Computational quotient space, 115 Results), 552 real numbers, 466 R (software), 546, 570–578 real-time algorithm, 512 Microsoft R Open, 578 recursion, 511 Rcpp, 577 reduced gradient, 207 RcppArmadillo, 577 reduced Hessian, 207 roxygen, 577 reduced rank regression problem, 432 RPy, 578 reducibility, 311, 334–336, 372 RStudio, 578 Markov chains, 444 Spotfire S+ (software), 578 reflection, 231–233 radix, 467 reflector, 233 random graph, 337 reflexive generalized inverse, 128 , 218–220 register, in computer processor, 485 BMvN distribution, 219 regression, 401–422, 426–431 computer generation Exercise 4.10:, regression variable selection, 427 221 regression, nonlinear, 201 correlation matrix, 441 regular graph, 329 Haar distribution, 219 regular matrix (see also diagonalizable Exercise 4.10:, 221 matrix), 149 normal, 218 regularization, 298, 405, 429 orthogonal Exercise 4.10:, 221 relative error, 484, 494, 526 rank Exercise 4.11:, 222 relative spacing, 470 Wishart, 122, 346, 421, 432 Reliable Computing, 493 Exercise 4.12:, 222 Replicated Computational Results random number generation, 440–442 (RCR), 552 random matrices Exercise 4.10:, 221 Reproducible R Toolkit, 578 random variable, 215 reproducible research, 551–552, 578 range of a matrix, 55 residue arithmetic, 493 rank deficiency, 100, 144 restarting, 525 rank determination, 532 reverse communication, 564 rank of a matrix, 99–122, 250, 431, 532 ρ(·) (spectral radius), 142 of idempotent matrix, 351 Richardson extrapolation, 510 rank-revealing QR, 250, 431 ridge regression, 270, 362, 405, 418, 429 statistical tests, 431–435 Exercise 9.11a:, 453 rank of an array, 5 right direct product, 95 rank reduction, 533 right inverse, 108 rank(·), 99 robustness (algorithm or software), 500 rank, linear independence, 99, 532 root of a function, 487 rank, number of dimensions, 5 root-free Cholesky, 255 rank-one decomposition, 164 Rosser test matrix, 550 rank-one update, 234, 285 rotation, 229–232, 236 rank-revealing QR, 250, 431, 532 rounding, 473 rate constant, 509 rounding error, 487, 495 rate of convergence, 509 , 111 rational fraction, 491 row rank, 99 Index 643 row space, 56 similarity transformation, 146–148, 313, row-major, 522, 539, 545 317 row-sum norm, 166 simple eigenvalue, 144 roxygen, 577 simple graph, 329 RPy, 578 simple matrix (see also diagonalizable RQ factorization, 247 matrix), 149 RStudio, 578 single precision, 472, 480 singular matrix, 100 S (software), 570 singular multivariate normal distribu- tion, 217, 432 Samelson inverse, 31 singular value, 162, 425, 532 sample variance, computing, 501 relation to eigenvalue, 163 saxpy, 12 singular value decomposition, 161–164, scalability, 506 320, 336, 425, 532 ScaLAPACK, 558 uniqueness, 163 scalar, 11 skew diagonal element, 57 scalar product, 23 skew diagonal matrix, 57 scaled matrix, 365 skew symmetric matrix, 56, 60 scaled vector, 49 skew upper triangular matrix, 58, 388 scaling of a vector or matrix, 269 skinny QR factorization, 247 scaling of an algorithm, 503, 506 smoothing matrix, 361, 418 Schatten p norm, 169 software testing, 548–551 Schur complement, 121, 412, 421 SOR (method), 279 Schur factorization, 147–148 span(·), 14, 21, 55 Schur norm (see also Frobenius norm), spanning set, 14, 21 167 of a cone, 44 SDP (semidefinite programming), 346 Spark (software system), 544 , 332 , 59, 259, 277, 523, 526, self-adjoint matrix (see also Hermitian 556, 557 matrix), 56 index-index-value, 548 semidefinite programming (SDP), 346 software, 557 seminorm, 25 CUDA, 560 semisimple eigenvalue, 144, 149 storage mode, 548 sequences of matrices, 171 spectral circle, 142 sequences of vectors, 32 spectral condition number, 268, 269, shape of matrix, 5 290 shearing transformation, 228 spectral decomposition, 155, 163 Sherman-Morrison formula, 285, 415 spectral norm, 166, 169 shifting eigenvalues, 310 spectral projector, 155 shrinkage, 405 spectral radius, 142, 166, 170, 278 side effect, 554 spectrum of a graph, 392 σ(·) (sign of permutation), 66, 86 spectrum of a matrix, 141–145 σ(·) (spectrum of matrix), 141 splitting extrapolation, 511 sign bit, 465 Spotfire S+ (software), 578 sign(·), 16 square root matrix, 160, 252, 254, 345 significand, 467 stability, 276, 500 similar canonical form, 149 standard deviation, 49, 364 similar matrices, 146 computing the standard deviation, similarity matrix, 369 501 644 Index

Standard Template Library, 569 Exercise 3.24:, 180 standards (see also specific standard), consistency test, 527, 551 460 Ericksen matrix, 549 stationary point of vector/matrix Exercise 12.8:, 582 functions, 198 Hilbert matrix, 548 statistical reference datasets (StRD), Matrix Market, 550 551 randomly generated data, 549 statlib, xiv, 617 Exercise 11.5:, 536 steepest descent, 198, 199 Rosser matrix, 550 Stiefel manifold, 133 StRD (statistical reference datasets), stiff data, 502 551 stochastic matrix, 377 Wilkinson matrix, 550 stochastic process, 442–450 Exercise 12.9:, 582 stopping criterion, 508 Wilkinson’s polynomial, 499 storage mode, for matrices, 546–548 testable hypothesis, 410 storage unit, 464, 467, 480 testing software, 548–551 Strassen algorithm, 529–531 thread, 544 StRD (statistical reference datasets), Tikhonov regularization, 299, 429 551 time series, 447–450 stride, 522, 539, 554 variance-covariance matrix, 383, 449 string, character, 461 , 382, 449 strongly connected graph, 334 circulant matrix, 383 submatrix, 61, 79 inverse of, 383 subspace of vector space, 17 Exercise 8.12:, 395 successive overrelaxation, 279 Exercise 12.12:, 583 summation, 485, 486 total least squares, 299, 405 summing vector, 34 tr(·), 65 Sun ONE Studio Fortran 95, 493 , 65 superlinear convergence, 509 derivative of, 195 SVD (singular value decomposition), of Cayley product, 87 161–164, 320, 336, 425, 532 of idempotent matrix, 351 uniqueness, 163 of inner product, 92 sweep operator, 413 of Kronecker product, 96 Sylvester’s law of nullity, 117 of matrix inner product, 97 symmetric matrix, 56, 60, 112, 153–160, of outer product, 92 338–343 relation to eigenvalues, 141 eingenvalues/vectors, 153–160 trace norm, 169 equivalent forms, 112 transition matrix, 443 inverse of, 120 translation transformation, 232 summary of properties, 338 transpose, 59 symmetric pair, 319 determinant of, 70 symmetric storage mode, 61, 547 generalized inverse of, 124 inverse of, 107 Taylor series, 188, 198 norm of, 164 Template Numerical Toolkit, 569 of Cayley product of matrices, 76 tensor, 5 of Kronecker product, 95 term-document matrix, 336 of partitioned matrices, 63 test problems for algorithms or software, of sum of matrices, 63 527, 548–551 trace of, 65 Index 645 trapezoidal matrix, 58, 241, 246 vech(·), 61 triangle inequality, 25, 164 vector, 4 Exercise 2.11b:, 52 centered vector, 48 triangular matrix, 57, 84, 240 mean vector, 35 determinant of, 70 normal vector, 34 inverse of, 121 normalized vector, 31 multiplication, 78 null vector, 15 , 58 one vector, 15, 34 triple scalar product Exercise 2.19c:, 54 “row vector”, 89 triple vector product Exercise 2.19d:, 54 scaled vector, 49 truncation error, 42, 99, 498 sign vector, 16 twos-complement representation, 465, summing vector, 15, 34 483 unit vector, 16 type 2 matrix, 58, 383 zero vector, 15 vector derivative, 183–220 ulp (“unit in the last place”), 471 vector processing, 557 underdetermined linear system, 123 vector space, 13–15, 17–23, 55, 64, 126, underflow, in computer operations, 470, 127 487 basis, 21–23 Unicode, 461 definition, 13 union of vector spaces, 18 dimension, 14 unit in the last place (ulp), 471 direct product, 20 unit roundoff, 470 direct sum, 18–20 unit vector, 16, 22, 36, 76 direct sum decomposition, 19 unitarily diagonalizable, 147, 343, 387 essentially disjoint, 15 unitarily similar, 146 intersection, 18 , 132 null vector space, 13 unrolling do-loop, 565 of matrices, 64 updating a solution, 285, 293, 415–417 order, 15 regression computations, 415–417 set operations, 17 upper Hessenberg form, 59, 316 subspace, 17 upper triangular matrix, 57 union, 18 usual norm (see also Frobenius norm), vector subspace, 17 167 vectorized processor, 508 vertex of a graph, 329 V(·) (variance operator), 49, 216 volume as a determinant, 73, 217 V(·) (vector space), 21, 55 , 379 Fourier matrix, 384 weighted graph, 329 variable metric method, 199 weighted least squares, 414 variable selection, 427 with equality constraints Exer- variance, computing, 501 cise 9.4d:, 451 variance-covariance matrix, 216, 219 weighted norm, 28, 93 Kronecker structure, 219, 419 Wilk’s Λ, 421 positive definite approximation, 436 Wilkinson matrix, 550 sample, 365, 422 Wishart distribution, 122, 346 vec(·), 61 Exercise 4.12:, 222 vec-permutation matrix, 82 Woodbury formula, 285, 415 vecdiag(·), 56 word, computer, 464, 467, 480 646 Index

XDR (external data representation), Z-matrix, 393 481 , 76, 99 zero of a function, 487 Yule-Walker equations, 449 zero vector, 15