The Dissertation Committee for Nicholas Vieau Alger certifies that this is the approved version of the following dissertation: Data-Scalable Hessian Preconditioning for Distributed Parameter PDE-Constrained Inverse Problems Committee: Omar Ghattas, Supervisor Tan Bui-Thanh, Co-Supervisor George Biros Inderjit Dhillon Sergey Fomel J. Tinsley Oden Data-Scalable Hessian Preconditioning for Distributed Parameter PDE-Constrained Inverse Problems by Nicholas Vieau Alger DISSERTATION Presented to the Faculty of the Graduate School of The University of Texas at Austin in Partial Fulfillment of the Requirements for the Degree of DOCTOR OF PHILOSOPHY THE UNIVERSITY OF TEXAS AT AUSTIN May 2019 Dedicated to my parents, Jeff and Cindy, and my brothers, J.J. and Bob. Acknowledgments I thank my advisors Omar Ghattas and Tan Bui-Thanh. I am grateful to Omar and Tan for sharing their deep knowledge of inverse problems and optimization with me. Their intellectual courage|attempting to solve the right problems even if they are difficult—has inspired me. I also thank my committee, especially Sergey Fomel for our discussions about seismic imaging. My thinking has been influenced by interactions with many friends and colleagues in the CCGO and CSEO research groups, as well as other graduate students and researchers at the University of Texas at Austin. Much of what I know about inverse problems was learned through conversations and nature walks with James Martin. My research was greatly aided by many conversations with Aaron Myers, Vishwas Rao, and Pearl Flath. I credit Umberto Villa, Tan Bui-Thanh, and Omar Ghattas for their work on the augmented KKT preconditioner paper, of which they are co-authors. Additionally, I thank James Martin and Toby Isaac for helpful discussions regarding the augmented KKT preconditioner. The constant in the augmented KKT condition number bound was improved based on suggestions from an anonymous reviewer. I credit Vishwas Rao, Aaron Myers, Tan Bui-Thanh, and Omar Ghattas for their work on the adap- tive product-convolution scheme paper, of which they are co-authors. Additionally, I thank J.J. Alger, Benjamin Babcock, Joe Bishop, Andrew Potter, Georg Stadler, Um- berto Villa, and Hongyu Zhu for helpful discussions regarding the adaptive product- convolution scheme. This dissertation has benefited from comments and editing suggestions from J.J. Alger, Jeff Alger, Tan Bui-Thanh, Josh Chen, Omar Ghattas, Trevor Heise, Aaron Myers, Tom O'Leary, and Vishwas Rao. I especially thank J.J. for our many conversa- tions about this document during the writing process and for his editing suggestions, and Trevor for his extensive editing suggestions. iv Data-Scalable Hessian Preconditioning for Distributed Parameter PDE-Constrained Inverse Problems Publication No. Nicholas Vieau Alger, Ph.D. The University of Texas at Austin, 2019 Supervisors: Omar Ghattas Tan Bui-Thanh Hessian preconditioners are the key to efficient numerical solution of large-scale distributed parameter PDE-constrained inverse problems with highly informative data. Such inverse problems arise in many applications, yet solving them remains computationally costly. With existing methods, the computational cost depends on spectral properties of the Hessian which worsen as more informative data are used to reconstruct the unknown parameter field. The best case scenario from a scien- tific standpoint (lots of high-quality data) is therefore the worst case scenario from a computational standpoint (large computational cost). In this dissertation, we argue that the best way to overcome this predicament is to build data-scalable Hessian/KKT preconditioners|preconditioners that perform well even if the data are highly informative about the parameter. We present a novel data-scalable KKT preconditioner for a diffusion inverse problem, a novel data- scalable Hessian preconditioner for an advection inverse problem, and a novel data- scalable domain decomposition preconditioner for an auxiliary operator that arises in connection with KKT preconditioning for a wave inverse problem. Our novel preconditioners outperform existing preconditioners in all three cases: they are robust to large numbers of observations in the diffusion inverse problem, large Peclet numbers in the advection inverse problem, and high wave frequencies in the wave inverse problem. v Table of Contents Acknowledgments iv Abstractv Chapter 1. Introduction1 1.1 Model problems . .2 1.1.1 Diffusion model problem . .3 1.1.2 Advection model problem . .4 1.1.3 Wave model problem . .5 1.1.4 Operators preconditioned . .6 1.2 Setting and notation . .7 1.3 The inverse problem . .8 1.4 Information, the Hessian, and the KKT operator . .9 1.5 Regularization preconditioning is not good enough . 11 1.6 Summary of the dissertation . 12 1.6.1 Background summary . 12 1.6.2 Hessian preconditioners summary . 14 1.6.3 Appendix summary . 18 Chapter 2. Inverse Problem Frameworks 19 2.1 Ill-posedness . 20 2.1.1 Ill-posedness examples . 21 2.2 Deterministic framework: Occam's razor . 23 2.3 Bayesian framework . 25 2.4 Connections between the frameworks . 26 2.5 Discussion of the quadratic/Gaussian assumptions . 27 2.6 Local quadratic/Gaussian models . 27 2.7 Choice of regularization/prior in practice . 30 2.7.1 Over- and under-regularization . 31 2.7.2 Motivation for smoothing regularization/priors . 33 Chapter 3. Deterministic Solution Methods 35 3.1 Ill-conditioning hobbles first-order methods . 36 3.1.1 Climbing an N-dimensional mountain: \hypercube paths" . 37 3.1.2 Zig-zag paths vs. hypercube paths . 39 3.2 Second-order methods . 44 3.2.1 Reduced space . 46 3.2.2 Full space . 46 3.3 Krylov methods prefer clustered eigenvalues . 48 3.4 Summary . 50 Chapter 4. Spectrum of the Hessian and Information 51 4.1 Simultaneously factoring the prior and posterior . 51 4.2 Informativeness of the data . 53 vi Chapter 5. Hessian and KKT Facts 57 5.1 Prerequisite partial derivative operators . 58 5.2 Computing derivatives of J ........................ 58 5.3 Formulas for other operators . 60 5.4 Connections among operators . 60 Chapter 6. Existing Hessian Solvers, Preconditioners, and Approxi- mations 63 6.1 Desired properties for SPAs . 63 6.2 Dense factorization of the Hessian . 65 6.3 Sparse-direct factorization of the KKT matrix . 65 6.4 Multigrid . 66 6.5 Low-rank approximation and regularization preconditioning . 67 6.6 Adjoint Schur complement and block scaling preconditioners . 69 6.7 Convolution interpolation . 70 6.8 Hierarchical matrices . 71 6.9 Other methods . 74 Chapter 7. Augmented Lagrangian KKT preconditioner 75 7.1 Overview . 76 7.2 Commentary on solving the preconditioner subsystems . 77 7.3 Derivation of the preconditioner . 78 7.4 Abstract analysis of the preconditioner . 81 7.4.1 Prerequisites . 81 7.4.1.1 Preconditioned KKT operator . 81 7.4.1.2 Arithmetic and geometric mean assumptions . 82 7.4.1.3 Brezzi theory for well posedness of saddle point systems 83 7.4.2 Bound on the condition number of the preconditioned KKT op- erator . 84 7.4.2.1 Bounds on X and Z ................... 84 7.4.2.2 Well posedness, continuity, and conditioning of the pre- conditioned KKT operator, E .............. 86 7.5 Spectral filtering and appropriate regularization assumptions . 88 7.5.1 Spectral filtering regularization . 88 7.5.2 Appropriate regularization assumptions . 89 7.6 Analysis of the source inversion problem with spectral filtering regular- ization . 91 7.7 Numerical results . 93 7.7.1 Convergence comparison . 95 7.7.2 Mesh scalability . 97 7.7.3 Regularization and data scalability . 98 Chapter 8. Adaptive Product-Convolution Approximation 100 8.1 Background . 102 8.1.1 Overview of results . 102 8.1.2 Setting and notation . 105 8.1.3 Product-convolution vs. convolution-product . 106 8.2 The adaptive product-convolution approximation . 107 8.2.1 Overview of the approximation . 108 8.2.2 Adaptive grid structure . 110 8.2.3 Adaptive refinement algorithm . 111 8.2.4 Harmonic weighting functions . 112 vii 8.2.5 Extended impulse response functions . 113 8.2.6 Randomized a-posteriori error estimator . 114 8.2.7 Anisotropic refinement: choosing the subdivision direction . 115 8.2.8 Construction cost . 116 8.3 Using the product-convolution approximation . 118 8.3.1 Computing matrix entries of A .................. 118 8.3.2 Applying A or A∗ to vectors . 118 8.3.3 Applying blocks of A or A∗ toe vectors . 119 8.3.4 Conversione to hierarchicale matrix format . 120 8.4 Theory . .e . .e . 121 8.5 Numerical Results . 126 Chapter 9. Domain Decomposition Wave Preconditioner 132 9.1 Algebraic impedance-to-impedance interface conditions . 133 H H 1 9.2 Replacing A with A in B YB + ρA WL− A .............. 135 9.3 Expanded system . 136 9.4 Reordering and block-factoringe the expanded system . 136 9.5 Interface Schur complement preconditioner . 138 9.5.1 PML truncation . 139 9.5.2 Product-convolution approximation . 140 9.5.3 Expanded truncated system . 143 9.5.4 Hierarchical matrix approximation of Mtrunc and definition of S 144 9.6 Setup and solve algorithms . 145 9.7 Numerical Results . .c . .b . 146 Chapter 10. Conclusion 150 Appendices 153 Appendix A. Bayesian Solution Methods 154 A.1 Throwing darts at an ellipsoid . 155 A.2 Sampling methods . 156 A.3 Required Hessian operations . 157 A.3.1 Rational matrix function approximation . 159 A.3.2 Inverse square root . 160 A.3.3 Determinant ratio . 161 Appendix B. Dimension dependence of sparse-direct methods 162 Appendix C. Additional Algorithms 166 Appendix D. Additional Proofs 167 Appendix E. Additional Numerical Results 171 E.1 Spatially varying blur . 171 E.2 Poisson interface Schur complement . 172 Bibliography 178 viii Chapter 1 Introduction In distributed parameter inverse problems governed by partial differential equa- tions (PDEs), one seeks to reconstruct an unknown spatially varying parameter field from measurements (\data") of a state variable that depend on the parameter implic- itly through the solution of a PDE.1 Although such inverse problems arise in a wide variety of applications, solving them remains computationally costly since a large number of PDEs must be solved in the process.